[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:53:59 dist-test-slave-pzsx krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:53:59 dist-test-slave-pzsx krb5kdc[395](info): set up 2 sockets
Sep 29 00:53:59 dist-test-slave-pzsx krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 00:54:00.309743 388 mini_kdc.cc:121] Time spent starting KDC: real 2.370s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:00 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:00.475876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:00.800958 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:00.801270 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:00.801435 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:00.809118 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:00.809228 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:00.809331 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:00.809432 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:00.809494 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:00.819439 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:00.820910 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:00.822266 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:00.834604 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.834659 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:00.834702 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:00.835589 408 server_base.cc:1034] running on GCE node
I20260929 00:54:00.836306 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:00.837685 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:00.838877 408 hybrid_clock.cc:648] HybridClock initialized: now 1790643240838809 us; error 63 us; skew 500 ppm
I20260929 00:54:00.839232 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:00 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:00.842217 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:00.843631 408 webserver.cc:460] Webserver started at http://127.0.97.62:41129/ using document root <none> and password file <none>
I20260929 00:54:00.844314 408 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:00.844457 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:00.844810 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:00.848110 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "097e8ed4faf6408f980ce678bb6b5ca6"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-pzsx"
server_key: "4f774d9b36a216d08e9a23012d43376f"
server_key_iv: "5272567a50047cf7a0de1bdbf1fc23af"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.849020 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "097e8ed4faf6408f980ce678bb6b5ca6"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-pzsx"
server_key: "4f774d9b36a216d08e9a23012d43376f"
server_key_iv: "5272567a50047cf7a0de1bdbf1fc23af"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.854976 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:00.858556 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:00.859934 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:00.860121 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "097e8ed4faf6408f980ce678bb6b5ca6"
format_stamp: "Formatted at 2026-09-29 00:54:00 on dist-test-slave-pzsx"
server_key: "4f774d9b36a216d08e9a23012d43376f"
server_key_iv: "5272567a50047cf7a0de1bdbf1fc23af"
server_key_version: "encryptionkey@0"
I20260929 00:54:00.860323 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:00.878057 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:00.882589 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:00.882920 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:00.901142 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37801
I20260929 00:54:00.901160 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37801 every 8 connection(s)
I20260929 00:54:00.902608 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:00.906725 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:00.912436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 408
I20260929 00:54:00.912806 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:00.914597 388 external_mini_cluster.cc:1428] Setting key 655d67b11c883cfaa4b0092b07691d45
I20260929 00:54:00.916465 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6: Bootstrap starting.
I20260929 00:54:00.920792 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:00.922187 475 log.cc:824] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:00.924916 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6: No bootstrap required, opened a new log
I20260929 00:54:00.933744 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } }
I20260929 00:54:00.934145 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:00.934264 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097e8ed4faf6408f980ce678bb6b5ca6, State: Initialized, Role: FOLLOWER
I20260929 00:54:00.934825 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } }
I20260929 00:54:00.935065 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:00.935204 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:00.935367 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:00.937719 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } }
I20260929 00:54:00.938195 475 leader_election.cc:302] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097e8ed4faf6408f980ce678bb6b5ca6; no voters:
I20260929 00:54:00.938846 475 leader_election.cc:288] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:00.938968 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:00.940012 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 097e8ed4faf6408f980ce678bb6b5ca6, State: Running, Role: LEADER
I20260929 00:54:00.940764 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } }
I20260929 00:54:00.942058 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 00:54:00 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:00.943696 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097e8ed4faf6408f980ce678bb6b5ca6. Latest consensus state: current_term: 1 leader_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } } }
I20260929 00:54:00.944098 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:00.944700 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097e8ed4faf6408f980ce678bb6b5ca6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37801 } } }
I20260929 00:54:00.945425 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:00.953483 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:00.959170 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:00.959776 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:00.918351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.97.62:37801)
0929 00:54:00.919278 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:00.919289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:00.920496 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 00:54:00.929463 (+ 8967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:00.929484 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:00.929856 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:00.930285 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:00.933241 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.934794 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.936677 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:00.937345 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:00.937552 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:00.945918 (+ 8366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:00.945932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:00.945957 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:00.946028 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:00.951686 (+ 5658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.953188 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.953201 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.953211 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.953491 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:00.954120 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:00.954142 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:00.954145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:00.954875 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:00.957537 (+ 2662us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:00.957841 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 00:54:00.958283 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":476,"threads_started":1}
I20260929 00:54:00.971208 485 catalog_manager.cc:1347] Generated new cluster ID: 5c94856cb094497488275d57619caccf
I20260929 00:54:00.971333 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:00.997924 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:00.999771 485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:01.008313 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 097e8ed4faf6408f980ce678bb6b5ca6: Generated new TSK 0
I20260929 00:54:01.009351 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:01.057766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:01.387421 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.387759 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.387928 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.395139 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.395277 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.395460 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:01.405354 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:01.406646 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.407934 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:01.420977 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.421029 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.421049 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.421780 501 server_base.cc:1034] running on GCE node
I20260929 00:54:01.422504 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.423398 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.424598 501 hybrid_clock.cc:648] HybridClock initialized: now 1790643241424492 us; error 95 us; skew 500 ppm
I20260929 00:54:01.424917 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:01 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.427944 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:01.429559 501 webserver.cc:460] Webserver started at http://127.0.97.1:34399/ using document root <none> and password file <none>
I20260929 00:54:01.430202 501 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.430300 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.430563 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:01.433809 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "d6492284d08340678a84ea9f19a393a7"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "f511c1284c7e48f6adc1cf215ec045e1"
server_key_iv: "59116e9cb50c29167edad8410017663b"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.434543 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d6492284d08340678a84ea9f19a393a7"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "f511c1284c7e48f6adc1cf215ec045e1"
server_key_iv: "59116e9cb50c29167edad8410017663b"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.439370 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 00:54:01.442595 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.444053 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:01.444221 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "d6492284d08340678a84ea9f19a393a7"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "f511c1284c7e48f6adc1cf215ec045e1"
server_key_iv: "59116e9cb50c29167edad8410017663b"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.444444 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.463325 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.468073 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.468366 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.469338 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:01.470600 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:01.470667 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.470827 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:01.470888 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.503365 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40815
I20260929 00:54:01.503400 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40815 every 8 connection(s)
I20260929 00:54:01.504721 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:01.505255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 501
I20260929 00:54:01.505473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:01.506219 388 external_mini_cluster.cc:1428] Setting key df3beb02665462dc87ebe50b74ea6fcb
Sep 29 00:54:01 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:54:01.524952 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.507430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48025 (local address 127.0.97.62:37801)
0929 00:54:01.507934 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.507940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.509051 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 00:54:01.510811 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.510816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.510901 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.511012 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.512455 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.513246 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.514751 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.515131 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.515261 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.518910 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.518916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.518919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.518948 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:01.521925 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.522710 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.522715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.522720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.522832 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:01.523242 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:01.523248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:01.523252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:01.523510 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:01.523662 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:01.524475 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 00:54:01.524707 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
I20260929 00:54:01.526950 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.508113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37801 (local address 127.0.97.1:48025)
0929 00:54:01.508915 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.508953 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.510557 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.511262 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.511270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.511747 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.512299 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.512316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.513402 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.513407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.514602 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.514613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.515348 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.515356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.515584 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.516264 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.516289 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.518699 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:01.522061 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.522071 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.522089 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.522584 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.522942 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:01.522947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:01.522950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:01.523129 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:01.523637 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:01.523644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:01.524040 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 00:54:01.524384 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 00:54:01.525013 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":418,"threads_started":1}
I20260929 00:54:01.528534 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37801
I20260929 00:54:01.528839 629 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:01.529562 629 heartbeater.cc:511] Master 127.0.97.62:37801 requested a full tablet report, sending...
I20260929 00:54:01.531771 440 ts_manager.cc:194] Registered new tserver with Master: d6492284d08340678a84ea9f19a393a7 (127.0.97.1:40815)
I20260929 00:54:01.533981 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:01.545568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:01.859274 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:01.859591 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:01.859759 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:01.866905 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:01.867009 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:01.867187 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:01.877373 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:01.878712 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:01.880111 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:01.894358 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.894302 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:01.894336 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:01.895494 635 server_base.cc:1034] running on GCE node
I20260929 00:54:01.896095 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:01.896994 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:01.898185 635 hybrid_clock.cc:648] HybridClock initialized: now 1790643241898123 us; error 54 us; skew 500 ppm
I20260929 00:54:01.898430 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:01 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:01.901592 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:01.910123 635 webserver.cc:460] Webserver started at http://127.0.97.2:43341/ using document root <none> and password file <none>
I20260929 00:54:01.910828 635 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:01.910974 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:01.911273 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:01.914758 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "fdb27220a255466fbaa628b026b02576"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "77ef454c5f2518ef71d64430e474edb7"
server_key_iv: "d1846ce80eb98fba2265b64be475ec25"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.915566 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fdb27220a255466fbaa628b026b02576"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "77ef454c5f2518ef71d64430e474edb7"
server_key_iv: "d1846ce80eb98fba2265b64be475ec25"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.922009 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 00:54:01.926223 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.927614 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:01.927816 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "fdb27220a255466fbaa628b026b02576"
format_stamp: "Formatted at 2026-09-29 00:54:01 on dist-test-slave-pzsx"
server_key: "77ef454c5f2518ef71d64430e474edb7"
server_key_iv: "d1846ce80eb98fba2265b64be475ec25"
server_key_version: "encryptionkey@0"
I20260929 00:54:01.928038 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:01.938834 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:01.943768 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:01.944034 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:01.944962 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:01.946277 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:01.946345 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.946504 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:01.946561 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:01.981698 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40127
I20260929 00:54:01.981738 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40127 every 8 connection(s)
I20260929 00:54:01.982925 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:01.991688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 635
I20260929 00:54:01.991873 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:01.992304 388 external_mini_cluster.cc:1428] Setting key 5dc56f66750f32c55bfc6e1ace5ec79d
Sep 29 00:54:01 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:02.003408 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.985410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56609 (local address 127.0.97.62:37801)
0929 00:54:01.985585 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:01.985590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:01.986683 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 00:54:01.988972 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:01.988978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:01.989093 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:01.989198 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:01.990930 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.991854 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.993617 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.994120 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.994268 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.997593 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:01.997602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:01.997610 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:01.997654 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.000299 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.001241 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.001244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.001247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.001358 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.001827 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.001830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.001832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.002057 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.002155 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.003010 (+ 855us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.003169 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:54:02.005105 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:01.985859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37801 (local address 127.0.97.2:56609)
0929 00:54:01.986519 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:01.986561 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:01.988623 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:01.989382 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:01.989390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:01.989866 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:01.990790 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.990807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.992031 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.992035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.993487 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:01.993498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:01.994352 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:01.994356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:01.994555 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:01.995221 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:01.995242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:01.997368 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:02.000430 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:02.000442 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:02.000470 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:02.001126 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:02.001470 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:02.001477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:02.001484 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:02.001717 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:02.002174 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:02.002185 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:02.002635 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 00:54:02.002940 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 00:54:02.003536 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260929 00:54:02.006654 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37801
I20260929 00:54:02.006955 763 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:02.007644 763 heartbeater.cc:511] Master 127.0.97.62:37801 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:02.009212 440 ts_manager.cc:194] Registered new tserver with Master: fdb27220a255466fbaa628b026b02576 (127.0.97.2:40127)
I20260929 00:54:02.010048 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:02.031939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:02.358366 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:02.358651 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:02.358757 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:02.366225 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:02.366328 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:02.366437 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:02.375856 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37801
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:02.377134 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:02.378381 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:02.391692 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.391671 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:02.391743 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:02.392786 768 server_base.cc:1034] running on GCE node
I20260929 00:54:02.393400 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:02.394356 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:02.395582 768 hybrid_clock.cc:648] HybridClock initialized: now 1790643242395540 us; error 58 us; skew 500 ppm
I20260929 00:54:02.395892 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:02.399359 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:02.400895 768 webserver.cc:460] Webserver started at http://127.0.97.3:44315/ using document root <none> and password file <none>
I20260929 00:54:02.401587 768 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:02.401724 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:02.402037 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:02.405041 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "a5fbdc2ae0bf4029bd27c369a7443691"
format_stamp: "Formatted at 2026-09-29 00:54:02 on dist-test-slave-pzsx"
server_key: "19c8d05a95ef9d191f4a3d10b3db3536"
server_key_iv: "c53dd3e8502e3f5d574dd24d07b9ad5a"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.405844 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a5fbdc2ae0bf4029bd27c369a7443691"
format_stamp: "Formatted at 2026-09-29 00:54:02 on dist-test-slave-pzsx"
server_key: "19c8d05a95ef9d191f4a3d10b3db3536"
server_key_iv: "c53dd3e8502e3f5d574dd24d07b9ad5a"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.410868 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 00:54:02.414137 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.415416 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:02.415608 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "a5fbdc2ae0bf4029bd27c369a7443691"
format_stamp: "Formatted at 2026-09-29 00:54:02 on dist-test-slave-pzsx"
server_key: "19c8d05a95ef9d191f4a3d10b3db3536"
server_key_iv: "c53dd3e8502e3f5d574dd24d07b9ad5a"
server_key_version: "encryptionkey@0"
I20260929 00:54:02.415813 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:02.439429 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:02.443929 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:02.444221 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:02.445176 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:02.446805 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:02.446952 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 00:54:02.447084 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:02.447149 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:02.481961 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36027
I20260929 00:54:02.482018 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36027 every 8 connection(s)
I20260929 00:54:02.483227 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:02.490908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 768
I20260929 00:54:02.491114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:02.491534 388 external_mini_cluster.cc:1428] Setting key 33e2fa70bfc5b7333560173a99f11f1c
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:02.502918 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.485704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42361 (local address 127.0.97.62:37801)
0929 00:54:02.485903 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.485909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.487416 (+ 1507us) server_negotiation.cc:407] Connection header received
0929 00:54:02.489334 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.489338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.489400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.489486 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.491049 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.491865 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.494079 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.494651 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.494803 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.497942 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.497946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.497949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.497978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.500284 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.500949 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.500952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.500954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.501068 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.501397 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.501400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.501402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.501629 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.501734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.502388 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.502590 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:54:02.503531 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.486237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37801 (local address 127.0.97.3:42361)
0929 00:54:02.487250 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:02.487287 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:02.489094 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:02.489687 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:02.489694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:02.490163 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:02.490845 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:02.490870 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.492000 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.492006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:02.493915 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:02.493932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.494864 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.494868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:02.495047 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.495713 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:02.495734 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:02.497774 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:02.500394 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:02.500401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:02.500411 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:02.500849 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:02.501154 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:02.501157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:02.501159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:02.501308 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:02.501727 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:02.501733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:02.501996 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 00:54:02.502215 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 00:54:02.502545 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":623,"threads_started":1}
I20260929 00:54:02.504916 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37801
I20260929 00:54:02.505203 896 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:02.506011 896 heartbeater.cc:511] Master 127.0.97.62:37801 requested a full tablet report, sending...
I20260929 00:54:02.507522 440 ts_manager.cc:194] Registered new tserver with Master: a5fbdc2ae0bf4029bd27c369a7443691 (127.0.97.3:36027)
I20260929 00:54:02.508325 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42361
I20260929 00:54:02.519591 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:02.536901 629 heartbeater.cc:503] Master 127.0.97.62:37801 was elected leader, sending a full tablet report...
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:02.563582 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.548126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.0.97.1:40815)
0929 00:54:02.548813 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.548819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.548848 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:54:02.549025 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.549032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.549250 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.549374 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.550727 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.551877 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.553405 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.553998 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.554097 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.557782 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.557792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.557806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.557835 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.560856 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.561560 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.561571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.561577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.561748 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.562222 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.562226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.562228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.562537 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.562749 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.562967 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.563212 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":386,"threads_started":1}
W20260929 00:54:02.564796 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45122
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:02.606144 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.592961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.0.97.1:40815)
0929 00:54:02.593183 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.593189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.593422 (+ 233us) server_negotiation.cc:407] Connection header received
0929 00:54:02.593635 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.593640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.593726 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.593835 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.595099 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.595784 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.596984 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.597424 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.597562 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.601542 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.601547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.601550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.601579 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.603776 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.604375 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.604386 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.604389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.604492 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.604891 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.604895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.604899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.605122 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.605344 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.605643 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.605846 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 00:54:02.607553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 501
I20260929 00:54:02.618842 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.608393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.0.97.1:40815)
0929 00:54:02.608567 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.608572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.608862 (+ 290us) server_negotiation.cc:407] Connection header received
0929 00:54:02.608985 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.608988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.609042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.609113 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.610137 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.610748 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.611856 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.612238 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.612322 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.614367 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.614372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.614374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.614400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.616823 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.617480 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.617483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.617485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.617605 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.617883 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.617886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.617888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.618106 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.618266 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.618464 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.618599 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:54:02.619565 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:02.700806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 501
I20260929 00:54:02.717268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 635
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:02.732379 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.718079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.0.97.2:40127)
0929 00:54:02.718591 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.718598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.718623 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:54:02.718681 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.718688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.718897 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.719020 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.720235 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.721285 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.722443 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.722955 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.723047 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.726926 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.726937 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.726950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.726986 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.729742 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.730353 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.730361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.730363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.730493 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.730879 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.730886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.730890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.731213 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.731557 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.731794 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.732048 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":271,"threads_started":1}
I20260929 00:54:02.733575 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:02.814057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 635
I20260929 00:54:02.836174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 768
Sep 29 00:54:02 dist-test-slave-pzsx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:02.851606 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:02.837038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.97.3:36027)
0929 00:54:02.837683 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:02.837690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:02.837720 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:54:02.837777 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:02.837784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:02.838002 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:02.838140 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:02.839244 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.840251 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.841713 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:02.842390 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:02.842486 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:02.846386 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:02.846396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:02.846411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:02.846452 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:02.849370 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.849949 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.849956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.849958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.850069 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:02.850361 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:02.850364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:02.850366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:02.850679 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:02.850889 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:02.851137 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 00:54:02.851326 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":331,"threads_started":1}
I20260929 00:54:02.853022 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:02.934091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 768
I20260929 00:54:02.950126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 408
I20260929 00:54:02.951248 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:03.049878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 408
2026-09-29T00:54:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5164 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:03 dist-test-slave-pzsx krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:03 dist-test-slave-pzsx krb5kdc[917](info): set up 2 sockets
Sep 29 00:54:03 dist-test-slave-pzsx krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 00:54:04.128518 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:04 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:04.309932 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:04.623605 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:04.623889 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:04.624037 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:04.631146 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:04.631254 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:04.631356 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:04.631456 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:04.631538 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:04.641321 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:04.642625 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:04.643913 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:04.655478 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:04.655448 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:04.655673 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:04.656796 930 server_base.cc:1034] running on GCE node
I20260929 00:54:04.657549 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:04.658802 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:04.659999 930 hybrid_clock.cc:648] HybridClock initialized: now 1790643244659952 us; error 45 us; skew 500 ppm
I20260929 00:54:04.660290 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:04 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:04.667534 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:04.669054 930 webserver.cc:460] Webserver started at http://127.0.97.62:37873/ using document root <none> and password file <none>
I20260929 00:54:04.669735 930 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:04.669878 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:04.670181 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:04.673122 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "545ebd6479504a7fa621f9cd7227dc5e"
format_stamp: "Formatted at 2026-09-29 00:54:04 on dist-test-slave-pzsx"
server_key: "e0d99dd693532f518524aa413a059662"
server_key_iv: "564062f0516514bf51a8555b4dd39935"
server_key_version: "encryptionkey@0"
I20260929 00:54:04.673846 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "545ebd6479504a7fa621f9cd7227dc5e"
format_stamp: "Formatted at 2026-09-29 00:54:04 on dist-test-slave-pzsx"
server_key: "e0d99dd693532f518524aa413a059662"
server_key_iv: "564062f0516514bf51a8555b4dd39935"
server_key_version: "encryptionkey@0"
I20260929 00:54:04.678783 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:04.682029 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:04.683255 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:54:04.683435 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "545ebd6479504a7fa621f9cd7227dc5e"
format_stamp: "Formatted at 2026-09-29 00:54:04 on dist-test-slave-pzsx"
server_key: "e0d99dd693532f518524aa413a059662"
server_key_iv: "564062f0516514bf51a8555b4dd39935"
server_key_version: "encryptionkey@0"
I20260929 00:54:04.683622 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:04.699810 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:04.703498 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:04.703769 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:04.721082 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43703
I20260929 00:54:04.721101 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43703 every 8 connection(s)
I20260929 00:54:04.722393 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:04.725973 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:04.726220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 930
I20260929 00:54:04.726449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:04.727276 388 external_mini_cluster.cc:1428] Setting key caf3b7fcb979057baf0e806b102fbc48
I20260929 00:54:04.739532 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e: Bootstrap starting.
Sep 29 00:54:04 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:04.742944 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:04.744124 997 log.cc:824] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:04.746804 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e: No bootstrap required, opened a new log
I20260929 00:54:04.749943 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:04.729292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.97.62:43703)
0929 00:54:04.730070 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:04.730080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:04.730119 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:54:04.732049 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:04.732072 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:04.732436 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:04.732821 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:04.734494 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:04.735713 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:04.737455 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:04.738068 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:04.738186 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:04.742588 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:04.742595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:04.742610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:04.742642 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:04.745641 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:04.746275 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:04.746283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:04.746291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:04.746452 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:04.746804 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:04.746809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:04.746813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:04.747352 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:04.747650 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:04.748097 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 00:54:04.748486 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":393,"threads_started":1}
I20260929 00:54:04.754990 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } }
I20260929 00:54:04.755330 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:04.755430 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545ebd6479504a7fa621f9cd7227dc5e, State: Initialized, Role: FOLLOWER
I20260929 00:54:04.755939 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } }
I20260929 00:54:04.756227 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:04.756376 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:04.756637 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:04.758949 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } }
I20260929 00:54:04.759428 997 leader_election.cc:302] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545ebd6479504a7fa621f9cd7227dc5e; no voters:
I20260929 00:54:04.760013 997 leader_election.cc:288] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:04.760154 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:04.760919 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [term 1 LEADER]: Becoming Leader. State: Replica: 545ebd6479504a7fa621f9cd7227dc5e, State: Running, Role: LEADER
I20260929 00:54:04.761478 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } }
I20260929 00:54:04.762072 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:04.764990 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545ebd6479504a7fa621f9cd7227dc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } } }
I20260929 00:54:04.764936 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545ebd6479504a7fa621f9cd7227dc5e. Latest consensus state: current_term: 1 leader_uuid: "545ebd6479504a7fa621f9cd7227dc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545ebd6479504a7fa621f9cd7227dc5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43703 } } }
I20260929 00:54:04.765481 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:04.765529 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:04.769366 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:04.774647 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:04.785808 1009 catalog_manager.cc:1347] Generated new cluster ID: 454589e0d18f43b091542ef41384eab9
I20260929 00:54:04.785939 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:04.821728 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:04.823086 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:04.830523 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545ebd6479504a7fa621f9cd7227dc5e: Generated new TSK 0
I20260929 00:54:04.831339 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:04.882328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:05.195549 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:05.195811 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:05.195904 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:05.202873 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:05.202982 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:05.203096 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:05.212395 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:05.213640 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:05.214898 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:05.227638 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:05.227619 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:05.227981 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:05.228778 1023 server_base.cc:1034] running on GCE node
I20260929 00:54:05.229369 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:05.230245 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:05.231463 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790643245231430 us; error 45 us; skew 500 ppm
I20260929 00:54:05.231767 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:05 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:05.234735 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:05.236074 1023 webserver.cc:460] Webserver started at http://127.0.97.1:38351/ using document root <none> and password file <none>
I20260929 00:54:05.236770 1023 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:05.236869 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:05.237123 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:05.239778 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "2fcbd21e591e4cd581193e71dd0b42dd"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "93af8c09e9eec5edc7cf5629d9378994"
server_key_iv: "5a7b864bc22fea45fd478c9a994fd507"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.240470 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2fcbd21e591e4cd581193e71dd0b42dd"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "93af8c09e9eec5edc7cf5629d9378994"
server_key_iv: "5a7b864bc22fea45fd478c9a994fd507"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.245325 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:54:05.248713 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:05.249986 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:54:05.250154 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "2fcbd21e591e4cd581193e71dd0b42dd"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "93af8c09e9eec5edc7cf5629d9378994"
server_key_iv: "5a7b864bc22fea45fd478c9a994fd507"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.250324 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:05.294064 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:05.297973 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:05.298297 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:05.299142 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:05.300347 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:05.300415 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:05.300549 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:05.300612 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 00:54:05.331837 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41985
I20260929 00:54:05.331887 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41985 every 8 connection(s)
I20260929 00:54:05.333175 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:05.339056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1023
I20260929 00:54:05.339270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:05.339839 388 external_mini_cluster.cc:1428] Setting key b985a623c3c4efc7ede57c03f31da3be
Sep 29 00:54:05 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:05.352782 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.335722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50895 (local address 127.0.97.62:43703)
0929 00:54:05.336147 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:05.336154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:05.337193 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 00:54:05.338899 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:05.338905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:05.338994 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:05.339112 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:05.340647 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.341385 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.342869 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.343273 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.343403 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:05.347007 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:05.347012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:05.347015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:05.347044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:05.349509 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:05.350427 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:05.350430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:05.350433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:05.350538 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:05.351009 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:05.351012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:05.351014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:05.351306 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:05.351448 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:05.352315 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 00:54:05.352469 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":197,"threads_started":1}
I20260929 00:54:05.354372 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.336330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43703 (local address 127.0.97.1:50895)
0929 00:54:05.337036 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:05.337080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:05.338639 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:05.339336 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:05.339368 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:05.339863 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:05.340439 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:05.340455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.341543 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.341548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:05.342728 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:05.342743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.343485 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.343494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:05.343725 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:05.344387 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:05.344407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:05.346808 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:05.349654 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:05.349665 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:05.349685 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:05.350312 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:05.350650 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:05.350657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:05.350663 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:05.350900 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:05.351454 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:05.351465 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:05.351916 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 00:54:05.352231 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 00:54:05.352872 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:05.355990 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43703
I20260929 00:54:05.356294 1151 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:05.357077 1151 heartbeater.cc:511] Master 127.0.97.62:43703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:54:05.359292 962 ts_manager.cc:194] Registered new tserver with Master: 2fcbd21e591e4cd581193e71dd0b42dd (127.0.97.1:41985)
I20260929 00:54:05.360765 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:05.379810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:05.690773 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:05.691097 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:05.691252 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:05.698266 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:05.698408 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:05.698589 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:05.708012 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:05.709327 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:05.710604 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:05.722967 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:05.723069 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:05.723701 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:05.724282 1157 server_base.cc:1034] running on GCE node
I20260929 00:54:05.725003 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:05.725912 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:05.727135 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790643245727077 us; error 50 us; skew 500 ppm
I20260929 00:54:05.727437 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:05 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:05.730507 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:05.732005 1157 webserver.cc:460] Webserver started at http://127.0.97.2:33675/ using document root <none> and password file <none>
I20260929 00:54:05.732901 1157 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:05.733028 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:05.733412 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:05.736085 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "3b225c86ca1647a184e3992561ca0149"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "16ebd2829fbe5bbe26910f71dd8ad56c"
server_key_iv: "a4778a203d3f2727e0bace325db4f438"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.736852 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b225c86ca1647a184e3992561ca0149"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "16ebd2829fbe5bbe26910f71dd8ad56c"
server_key_iv: "a4778a203d3f2727e0bace325db4f438"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.741724 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 00:54:05.745105 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:05.746348 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:05.746511 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "3b225c86ca1647a184e3992561ca0149"
format_stamp: "Formatted at 2026-09-29 00:54:05 on dist-test-slave-pzsx"
server_key: "16ebd2829fbe5bbe26910f71dd8ad56c"
server_key_iv: "a4778a203d3f2727e0bace325db4f438"
server_key_version: "encryptionkey@0"
I20260929 00:54:05.746695 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:05.763041 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:05.767319 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:05.767555 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:05.768373 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:05.769639 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:05.769707 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:05.769784 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:05.769831 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:05.801986 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40717
I20260929 00:54:05.802030 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40717 every 8 connection(s)
I20260929 00:54:05.803270 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:05.805024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1157
I20260929 00:54:05.805208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:05.805639 388 external_mini_cluster.cc:1428] Setting key 3cc1f8a8b59471940cbb255bf7a0ff46
Sep 29 00:54:05 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:05.822809 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.806119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53945 (local address 127.0.97.62:43703)
0929 00:54:05.807223 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:05.807228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:05.807727 (+ 499us) server_negotiation.cc:407] Connection header received
0929 00:54:05.809565 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:05.809570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:05.809650 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:05.809767 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:05.811190 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.811901 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.813462 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.813902 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.814032 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:05.817417 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:05.817422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:05.817425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:05.817454 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:05.819932 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:05.820615 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:05.820620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:05.820624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:05.820744 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:05.821163 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:05.821168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:05.821171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:05.821460 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:05.821615 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:05.822246 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 00:54:05.822464 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:54:05.823921 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:05.806774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43703 (local address 127.0.97.2:53945)
0929 00:54:05.807588 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:05.807631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:05.809319 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:05.809995 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:05.810003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:05.810494 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:05.811036 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:05.811053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.812046 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.812056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:05.813322 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:05.813333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:05.814117 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:05.814123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:05.814339 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:05.815038 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:05.815058 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:05.817189 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:05.820064 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:05.820073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:05.820089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:05.820473 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:05.820852 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:05.820857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:05.820861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:05.821052 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:05.821587 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:05.821595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:05.821912 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 00:54:05.822154 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 00:54:05.822762 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":360,"threads_started":1}
I20260929 00:54:05.825398 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43703
I20260929 00:54:05.825726 1285 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:05.826581 1285 heartbeater.cc:511] Master 127.0.97.62:43703 requested a full tablet report, sending...
I20260929 00:54:05.828017 962 ts_manager.cc:194] Registered new tserver with Master: 3b225c86ca1647a184e3992561ca0149 (127.0.97.2:40717)
I20260929 00:54:05.828851 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:05.843044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:06.156828 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:06.157101 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:06.157280 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:06.164485 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:06.164672 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:06.164860 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:06.174585 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:06.175863 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:06.177228 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:06.190212 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:06.190166 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:06.190124 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:06.191075 1290 server_base.cc:1034] running on GCE node
I20260929 00:54:06.191649 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:06.192603 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:06.193826 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790643246193783 us; error 54 us; skew 500 ppm
I20260929 00:54:06.194139 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:06.197153 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:06.198594 1290 webserver.cc:460] Webserver started at http://127.0.97.3:35811/ using document root <none> and password file <none>
I20260929 00:54:06.199273 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:06.199415 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:06.199721 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:06.202579 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "d42af61c09b54a3b9b0ce4715d56b627"
format_stamp: "Formatted at 2026-09-29 00:54:06 on dist-test-slave-pzsx"
server_key: "cd74ba51c59cb794ae9b024ed49d0124"
server_key_iv: "bb06054c4ac99c96b46b7f8de780841f"
server_key_version: "encryptionkey@0"
I20260929 00:54:06.203336 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d42af61c09b54a3b9b0ce4715d56b627"
format_stamp: "Formatted at 2026-09-29 00:54:06 on dist-test-slave-pzsx"
server_key: "cd74ba51c59cb794ae9b024ed49d0124"
server_key_iv: "bb06054c4ac99c96b46b7f8de780841f"
server_key_version: "encryptionkey@0"
I20260929 00:54:06.208379 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 00:54:06.211612 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:06.213047 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:06.213263 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "d42af61c09b54a3b9b0ce4715d56b627"
format_stamp: "Formatted at 2026-09-29 00:54:06 on dist-test-slave-pzsx"
server_key: "cd74ba51c59cb794ae9b024ed49d0124"
server_key_iv: "bb06054c4ac99c96b46b7f8de780841f"
server_key_version: "encryptionkey@0"
I20260929 00:54:06.213465 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:06.231160 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:06.234982 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:06.235262 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:06.236176 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:06.237627 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:06.237730 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:06.237884 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:06.237955 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:06.271202 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35945
I20260929 00:54:06.271255 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35945 every 8 connection(s)
I20260929 00:54:06.272470 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:06.278213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1290
I20260929 00:54:06.278398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:06.278903 388 external_mini_cluster.cc:1428] Setting key e75e907befb69dbe84b12864feb72b0e
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:06.292258 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.275362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44575 (local address 127.0.97.62:43703)
0929 00:54:06.275571 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.275577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.276848 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 00:54:06.278893 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.278897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.278990 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.279090 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.281013 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.282060 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.283481 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.283860 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.283963 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.287200 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.287205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.287208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.287241 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.289736 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.290368 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.290371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.290373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.290481 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.290816 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.290819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.290820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.291095 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.291211 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.291799 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.291946 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:54:06.293341 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.275957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43703 (local address 127.0.97.3:44575)
0929 00:54:06.276693 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.276736 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.278617 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.279350 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.279362 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.280019 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:06.280842 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.280865 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.282189 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.282192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.283367 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.283378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.284032 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.284037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.284215 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.284896 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:06.284921 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:06.287027 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:06.289853 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:06.289860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:06.289871 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:06.290264 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:06.290565 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:06.290568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:06.290570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:06.290724 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:06.291200 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:06.291205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:06.291483 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.291693 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.292030 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260929 00:54:06.295073 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43703
I20260929 00:54:06.295408 1418 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:06.296777 1418 heartbeater.cc:511] Master 127.0.97.62:43703 requested a full tablet report, sending...
I20260929 00:54:06.298257 962 ts_manager.cc:194] Registered new tserver with Master: d42af61c09b54a3b9b0ce4715d56b627 (127.0.97.3:35945)
I20260929 00:54:06.299027 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44575
I20260929 00:54:06.306243 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:06.324620 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.312069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.97.62:43703)
0929 00:54:06.312229 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.312233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.312741 (+ 508us) server_negotiation.cc:407] Connection header received
0929 00:54:06.312919 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.312924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.312993 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.313099 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.314472 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.315169 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.316392 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.316929 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.317055 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.319382 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.319388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.319392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.319432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.322086 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.322698 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.322706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.322710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.322830 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.323171 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.323176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.323179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.323469 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.323711 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.324038 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.324249 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:54:06.330330 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:06.337023 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:06.364183 1151 heartbeater.cc:503] Master 127.0.97.62:43703 was elected leader, sending a full tablet report...
I20260929 00:54:06.367501 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.349369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.0.97.3:35945)
0929 00:54:06.356443 (+ 7074us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.356454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.357241 (+ 787us) server_negotiation.cc:407] Connection header received
0929 00:54:06.357678 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.357685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.357910 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.358084 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:06.359751 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.361441 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.364624 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.366037 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.366152 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.366388 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.367002 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.367185 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6896,"thread_start_us":7869,"threads_started":1}
I20260929 00:54:06.367805 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.348636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.0.97.2:40717)
0929 00:54:06.356907 (+ 8271us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.356917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.356946 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:54:06.357041 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.357048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.357313 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.357494 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:06.359979 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.361242 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.365201 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.366301 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.366411 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.366627 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.367355 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.367510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8102,"thread_start_us":8010,"threads_started":1}
I20260929 00:54:06.367894 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.348530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40717 (local address 127.0.0.1:58946)
0929 00:54:06.356513 (+ 7983us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.356554 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.356747 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.357766 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.357773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.357800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:06.359794 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.359807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.361359 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.361367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.365018 (+ 3651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.365034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.366969 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.366975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.367158 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.367187 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.367312 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.367507 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7716,"mutex_wait_us":587,"thread_start_us":7432,"threads_started":1}
I20260929 00:54:06.368551 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.349051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35945 (local address 127.0.0.1:39562)
0929 00:54:06.357138 (+ 8087us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.357154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.357491 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.358371 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.358375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.358399 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:06.359549 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.359568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.361604 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.361608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.364018 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.364029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.366542 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.366550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.366734 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.366791 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.367983 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.368137 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7947,"mutex_wait_us":4,"thread_start_us":6342,"threads_started":1}
I20260929 00:54:06.369362 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.356983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41985 (local address 127.0.0.1:57956)
0929 00:54:06.360826 (+ 3843us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.360844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.360954 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.361474 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.361478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.361511 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:06.362355 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.362367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.364234 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.364241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.366279 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.366291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.368486 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.368517 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.368733 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.368764 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.368848 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.369002 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"mutex_wait_us":38,"thread_start_us":3524,"threads_started":1}
I20260929 00:54:06.369971 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.356766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.0.97.1:41985)
0929 00:54:06.358815 (+ 2049us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.358821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.360955 (+ 2134us) server_negotiation.cc:407] Connection header received
0929 00:54:06.361071 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.361078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.361284 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.361466 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:06.362558 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.364105 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.366622 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.368019 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.368136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.368345 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.369558 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.369720 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923,"thread_start_us":1825,"threads_started":1}
I20260929 00:54:06.372097 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 2eadc83c001b48b28e5d2d1c037b9526 (DEFAULT_TABLE table=test-table [id=faf2db97a7eb4846ad20bcc3b5e1d326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:06.372534 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 2eadc83c001b48b28e5d2d1c037b9526 (DEFAULT_TABLE table=test-table [id=faf2db97a7eb4846ad20bcc3b5e1d326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:06.373140 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 2eadc83c001b48b28e5d2d1c037b9526 (DEFAULT_TABLE table=test-table [id=faf2db97a7eb4846ad20bcc3b5e1d326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:06.373983 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eadc83c001b48b28e5d2d1c037b9526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:06.373984 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eadc83c001b48b28e5d2d1c037b9526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:06.374328 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eadc83c001b48b28e5d2d1c037b9526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:06.386790 1436 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Bootstrap starting.
I20260929 00:54:06.388751 1437 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Bootstrap starting.
I20260929 00:54:06.389490 1438 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Bootstrap starting.
I20260929 00:54:06.390527 1436 tablet_bootstrap.cc:654] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:06.392206 1436 log.cc:824] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:06.393239 1438 tablet_bootstrap.cc:654] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:06.394383 1438 log.cc:824] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:06.395190 1437 tablet_bootstrap.cc:654] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:06.396318 1436 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: No bootstrap required, opened a new log
I20260929 00:54:06.396656 1436 ts_tablet_manager.cc:1397] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 00:54:06.396696 1437 log.cc:824] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:06.397269 1438 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: No bootstrap required, opened a new log
I20260929 00:54:06.397524 1438 ts_tablet_manager.cc:1397] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 00:54:06.399029 1437 tablet_bootstrap.cc:492] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: No bootstrap required, opened a new log
I20260929 00:54:06.399257 1437 ts_tablet_manager.cc:1397] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260929 00:54:06.406141 1437 raft_consensus.cc:348] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.406489 1437 raft_consensus.cc:374] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:06.406599 1437 raft_consensus.cc:729] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42af61c09b54a3b9b0ce4715d56b627, State: Initialized, Role: FOLLOWER
I20260929 00:54:06.407207 1437 consensus_queue.cc:260] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.407419 1438 raft_consensus.cc:348] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.407883 1438 raft_consensus.cc:374] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:06.408007 1438 raft_consensus.cc:729] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b225c86ca1647a184e3992561ca0149, State: Initialized, Role: FOLLOWER
I20260929 00:54:06.408288 1418 heartbeater.cc:503] Master 127.0.97.62:43703 was elected leader, sending a full tablet report...
I20260929 00:54:06.408684 1438 consensus_queue.cc:260] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.408617 1436 raft_consensus.cc:348] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.409003 1436 raft_consensus.cc:374] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:06.409056 1437 ts_tablet_manager.cc:1428] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260929 00:54:06.409142 1436 raft_consensus.cc:729] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fcbd21e591e4cd581193e71dd0b42dd, State: Initialized, Role: FOLLOWER
I20260929 00:54:06.409834 1436 consensus_queue.cc:260] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.410238 1285 heartbeater.cc:503] Master 127.0.97.62:43703 was elected leader, sending a full tablet report...
I20260929 00:54:06.410796 1438 ts_tablet_manager.cc:1428] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 00:54:06.412890 1436 ts_tablet_manager.cc:1428] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
W20260929 00:54:06.524488 1419 tablet.cc:2369] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:06.557221 1286 tablet.cc:2369] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:06.585456 1152 tablet.cc:2369] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:06.612557 1444 raft_consensus.cc:482] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:06.612830 1444 raft_consensus.cc:504] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.614440 1444 leader_election.cc:288] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b225c86ca1647a184e3992561ca0149 (127.0.97.2:40717), d42af61c09b54a3b9b0ce4715d56b627 (127.0.97.3:35945)
I20260929 00:54:06.620869 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.614986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48687 (local address 127.0.97.3:35945)
0929 00:54:06.615218 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.615227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.615588 (+ 361us) server_negotiation.cc:407] Connection header received
0929 00:54:06.615810 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.615814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.615869 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.615962 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:06.616518 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.617444 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.619334 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.620069 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.620163 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.620235 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.620613 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.620696 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:06.621162 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.614726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35945 (local address 127.0.97.1:48687)
0929 00:54:06.615458 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.615478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.615668 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.615977 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.615981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.616003 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:06.616326 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.616334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.617569 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.617574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.619191 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.619202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.620265 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.620272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.620440 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.620480 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.620599 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.620728 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":2,"thread_start_us":383,"threads_started":1}
I20260929 00:54:06.621963 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.614886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37869 (local address 127.0.97.2:40717)
0929 00:54:06.615091 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.615100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.615831 (+ 731us) server_negotiation.cc:407] Connection header received
0929 00:54:06.616020 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.616027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.616102 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.616192 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:06.616814 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.617601 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.619366 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.620859 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.620977 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.621063 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.621613 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.621709 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 00:54:06.621984 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.614833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40717 (local address 127.0.97.1:37869)
0929 00:54:06.615708 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:06.615730 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:06.615875 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:06.616208 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:06.616212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:06.616237 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:06.616653 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.616664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.617730 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.617734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.619198 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:06.619208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.621215 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.621223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:06.621403 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.621425 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:06.621588 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 00:54:06.621711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":21,"thread_start_us":612,"threads_started":1}
I20260929 00:54:06.622352 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eadc83c001b48b28e5d2d1c037b9526" candidate_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42af61c09b54a3b9b0ce4715d56b627" is_pre_election: true
I20260929 00:54:06.622927 1373 raft_consensus.cc:2454] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fcbd21e591e4cd581193e71dd0b42dd in term 0.
I20260929 00:54:06.622979 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eadc83c001b48b28e5d2d1c037b9526" candidate_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b225c86ca1647a184e3992561ca0149" is_pre_election: true
I20260929 00:54:06.623507 1240 raft_consensus.cc:2454] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fcbd21e591e4cd581193e71dd0b42dd in term 0.
I20260929 00:54:06.623616 1042 leader_election.cc:302] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcbd21e591e4cd581193e71dd0b42dd, d42af61c09b54a3b9b0ce4715d56b627; no voters:
I20260929 00:54:06.624032 1444 raft_consensus.cc:2781] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:06.624182 1444 raft_consensus.cc:482] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:06.624351 1444 raft_consensus.cc:3037] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:06.626524 1444 raft_consensus.cc:504] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.627023 1444 leader_election.cc:288] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [CANDIDATE]: Term 1 election: Requested vote from peers 3b225c86ca1647a184e3992561ca0149 (127.0.97.2:40717), d42af61c09b54a3b9b0ce4715d56b627 (127.0.97.3:35945)
I20260929 00:54:06.627449 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eadc83c001b48b28e5d2d1c037b9526" candidate_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b225c86ca1647a184e3992561ca0149"
I20260929 00:54:06.627496 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eadc83c001b48b28e5d2d1c037b9526" candidate_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42af61c09b54a3b9b0ce4715d56b627"
I20260929 00:54:06.627622 1240 raft_consensus.cc:3037] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:06.627736 1373 raft_consensus.cc:3037] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:06.629798 1240 raft_consensus.cc:2454] T 2eadc83c001b48b28e5d2d1c037b9526 P 3b225c86ca1647a184e3992561ca0149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fcbd21e591e4cd581193e71dd0b42dd in term 1.
I20260929 00:54:06.629909 1373 raft_consensus.cc:2454] T 2eadc83c001b48b28e5d2d1c037b9526 P d42af61c09b54a3b9b0ce4715d56b627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fcbd21e591e4cd581193e71dd0b42dd in term 1.
I20260929 00:54:06.630175 1040 leader_election.cc:302] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcbd21e591e4cd581193e71dd0b42dd, 3b225c86ca1647a184e3992561ca0149; no voters:
I20260929 00:54:06.630445 1444 raft_consensus.cc:2781] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:06.631305 1444 raft_consensus.cc:686] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [term 1 LEADER]: Becoming Leader. State: Replica: 2fcbd21e591e4cd581193e71dd0b42dd, State: Running, Role: LEADER
I20260929 00:54:06.631759 1444 consensus_queue.cc:237] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } }
I20260929 00:54:06.636267 962 catalog_manager.cc:5495] T 2eadc83c001b48b28e5d2d1c037b9526 P 2fcbd21e591e4cd581193e71dd0b42dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fcbd21e591e4cd581193e71dd0b42dd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b225c86ca1647a184e3992561ca0149" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fcbd21e591e4cd581193e71dd0b42dd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42af61c09b54a3b9b0ce4715d56b627" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35945 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:06.663378 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.648668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.0.97.1:41985)
0929 00:54:06.648846 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.648850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.649359 (+ 509us) server_negotiation.cc:407] Connection header received
0929 00:54:06.649549 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.649555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.649635 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.649713 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.651166 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.651891 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.653130 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.653510 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.653634 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.657876 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.657888 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.657904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.657935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.660979 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.661533 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.661540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.661542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.661672 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.662033 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.662036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.662038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.662402 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.662618 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.662816 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.663055 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:06.706539 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.693303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.97.1:41985)
0929 00:54:06.693529 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.693538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.693836 (+ 298us) server_negotiation.cc:407] Connection header received
0929 00:54:06.694045 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.694049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.694110 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.694212 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.695442 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.696159 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.697459 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.697866 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.697992 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.701862 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.701866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.701870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.701901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.704107 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.704759 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.704764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.704767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.704872 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.705252 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.705256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.705259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.705491 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.705759 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.705988 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.706186 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:06.752719 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.739155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.97.1:41985)
0929 00:54:06.739395 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.739402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.739686 (+ 284us) server_negotiation.cc:407] Connection header received
0929 00:54:06.739868 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.739872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.739949 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.740027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.741379 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.742109 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.743252 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.743650 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.743738 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.747384 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.747389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.747393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.747423 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.750519 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.751080 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.751084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.751086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.751245 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.751524 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.751527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.751529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.751790 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.751971 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.752184 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.752338 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 00:54:06.754493 1066 tablet_service.cc:2908] Rejecting scan request for tablet 2eadc83c001b48b28e5d2d1c037b9526: Uninitialized: clean time has not yet been initialized
I20260929 00:54:06.757408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1023
I20260929 00:54:06.769070 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.758258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.0.97.1:41985)
0929 00:54:06.758433 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.758441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.758756 (+ 315us) server_negotiation.cc:407] Connection header received
0929 00:54:06.758866 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.758869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.758933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.759001 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.760004 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.760880 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.761954 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.762422 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.762510 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.764427 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.764431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.764434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.764460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.766933 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.767448 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.767452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.767454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.767571 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.767846 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.767848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.767850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.768058 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.768221 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.768438 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.768685 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:54:06.769687 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:06.871357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1023
I20260929 00:54:06.888650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1157
Sep 29 00:54:06 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:06.903005 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:06.889426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.0.97.2:40717)
0929 00:54:06.889609 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:06.889615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:06.889661 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:54:06.889870 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:06.889876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:06.890006 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:06.890120 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:06.891260 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.892030 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.893445 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:06.893838 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:06.893972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:06.897669 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:06.897681 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:06.897695 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:06.897725 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:06.900166 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.900932 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.900940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.900942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.901106 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:06.901499 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:06.901508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:06.901511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:06.901853 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:06.902137 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:06.902433 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 00:54:06.902655 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:54:06.903628 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:06.989575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1157
I20260929 00:54:07.005774 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1290
Sep 29 00:54:07 dist-test-slave-pzsx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:07.019697 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:07.006457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.97.3:35945)
0929 00:54:07.006618 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:07.006622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:07.006640 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:54:07.006769 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:07.006772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:07.006835 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:07.006916 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:07.008065 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:07.008841 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:07.010620 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:07.010980 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:07.011073 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:07.014858 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:07.014870 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:07.014886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:07.014918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:07.017582 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:07.018140 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:07.018147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:07.018149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:07.018276 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:07.018575 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:07.018578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:07.018581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:07.018878 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:07.019066 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:07.019286 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 00:54:07.019433 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:54:07.020284 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:07.105417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1290
I20260929 00:54:07.121119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 930
I20260929 00:54:07.122220 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:07.227252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 930
2026-09-29T00:54:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4177 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:07 dist-test-slave-pzsx krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:07 dist-test-slave-pzsx krb5kdc[1466](info): set up 2 sockets
Sep 29 00:54:07 dist-test-slave-pzsx krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 00:54:08.305487 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:08 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:08.482264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:08.804457 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:08.804800 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:08.804891 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:08.812001 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:08.812103 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:54:08.812144 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:08.812198 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:08.812240 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:08.812301 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:08.821920 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:08.823092 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:08.824363 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:08.835862 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:08.835804 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:08.835897 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:08.836395 1479 server_base.cc:1034] running on GCE node
I20260929 00:54:08.837230 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:08.838615 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:08.839830 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790643248839772 us; error 30 us; skew 500 ppm
I20260929 00:54:08.840164 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:08 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:08.843130 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:08.844645 1479 webserver.cc:460] Webserver started at http://127.0.97.62:44223/ using document root <none> and password file <none>
I20260929 00:54:08.845336 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:08.845463 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:08.845791 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:08.848757 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "5c25fbc6cba943f48a4654af2385622e"
format_stamp: "Formatted at 2026-09-29 00:54:08 on dist-test-slave-pzsx"
server_key: "372423af0d1a4d46dac73ae0f75c1ee7"
server_key_iv: "cc54734418ac2d3b280af1a6737bb502"
server_key_version: "encryptionkey@0"
I20260929 00:54:08.849584 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "5c25fbc6cba943f48a4654af2385622e"
format_stamp: "Formatted at 2026-09-29 00:54:08 on dist-test-slave-pzsx"
server_key: "372423af0d1a4d46dac73ae0f75c1ee7"
server_key_iv: "cc54734418ac2d3b280af1a6737bb502"
server_key_version: "encryptionkey@0"
I20260929 00:54:08.854795 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:08.858265 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:08.859524 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:08.859720 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "5c25fbc6cba943f48a4654af2385622e"
format_stamp: "Formatted at 2026-09-29 00:54:08 on dist-test-slave-pzsx"
server_key: "372423af0d1a4d46dac73ae0f75c1ee7"
server_key_iv: "cc54734418ac2d3b280af1a6737bb502"
server_key_version: "encryptionkey@0"
I20260929 00:54:08.859935 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:08.878753 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:08.882685 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:08.882987 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:08.904803 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38367
I20260929 00:54:08.904824 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38367 every 8 connection(s)
I20260929 00:54:08.906165 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:08.908126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1479
I20260929 00:54:08.908355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:08.909183 388 external_mini_cluster.cc:1428] Setting key 1d0e09852730676cf0ed10cadd7634cd
I20260929 00:54:08.910686 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 00:54:08 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:08.923746 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Bootstrap starting.
I20260929 00:54:08.927661 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:08.928944 1556 log.cc:824] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:08.932070 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: No bootstrap required, opened a new log
I20260929 00:54:08.932339 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:08.911167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.0.97.62:38367)
0929 00:54:08.911883 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:08.911898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:08.911948 (+ 50us) server_negotiation.cc:407] Connection header received
0929 00:54:08.913921 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:08.913947 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:08.914457 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:08.914887 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:08.916162 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:08.918293 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:08.919430 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:08.920436 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:08.920632 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:08.924393 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:08.924404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:08.924420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:08.924464 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:08.928197 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:08.928912 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:08.928932 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:08.928941 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:08.929131 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:08.929488 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:08.929496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:08.929501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:08.930054 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:08.930356 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:08.930708 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 00:54:08.931112 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":312,"threads_started":1}
I20260929 00:54:08.939313 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } }
I20260929 00:54:08.939620 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:08.939692 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c25fbc6cba943f48a4654af2385622e, State: Initialized, Role: FOLLOWER
I20260929 00:54:08.940137 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } }
I20260929 00:54:08.940336 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:08.940430 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:08.940613 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:08.942612 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } }
I20260929 00:54:08.943111 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c25fbc6cba943f48a4654af2385622e; no voters:
I20260929 00:54:08.943676 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:08.943842 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:08.944535 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [term 1 LEADER]: Becoming Leader. State: Replica: 5c25fbc6cba943f48a4654af2385622e, State: Running, Role: LEADER
I20260929 00:54:08.945063 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } }
I20260929 00:54:08.945727 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:08.948196 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c25fbc6cba943f48a4654af2385622e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } } }
I20260929 00:54:08.948927 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:08.948159 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c25fbc6cba943f48a4654af2385622e. Latest consensus state: current_term: 1 leader_uuid: "5c25fbc6cba943f48a4654af2385622e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c25fbc6cba943f48a4654af2385622e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38367 } } }
I20260929 00:54:08.949273 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:08.955147 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:08.960397 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:08.969735 1570 catalog_manager.cc:1347] Generated new cluster ID: 821047304a384d8bb3a683231c1e840c
I20260929 00:54:08.969857 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:08.979081 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:08.980963 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:08.988960 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Generated new TSK 0
I20260929 00:54:08.989939 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:09.036371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38367
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:09.357931 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:09.358219 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:09.358316 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:09.365798 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:09.365916 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:09.366006 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:09.366079 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:09.376267 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:09.377562 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:09.378820 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:09.393153 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:09.393121 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:09.393185 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:09.393901 1582 server_base.cc:1034] running on GCE node
I20260929 00:54:09.394552 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:09.395478 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:09.396720 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790643249396657 us; error 46 us; skew 500 ppm
I20260929 00:54:09.397013 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:09 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:09.400058 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:09.401517 1582 webserver.cc:460] Webserver started at http://127.0.97.1:38635/ using document root <none> and password file <none>
I20260929 00:54:09.402276 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:09.402407 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:09.402735 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:09.405637 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "6df63b96cb5a41af9966a81a589cd065"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "489ac3f03f356af9ce5876170f0dec82"
server_key_iv: "1d81f7e11e255604e38029c8cb218f55"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.406436 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6df63b96cb5a41af9966a81a589cd065"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "489ac3f03f356af9ce5876170f0dec82"
server_key_iv: "1d81f7e11e255604e38029c8cb218f55"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.411499 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 00:54:09.415107 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:09.416414 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:09.416616 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "6df63b96cb5a41af9966a81a589cd065"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "489ac3f03f356af9ce5876170f0dec82"
server_key_iv: "1d81f7e11e255604e38029c8cb218f55"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.416850 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:09.450587 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:09.455307 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:09.455590 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:09.458185 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:09.458297 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:09.458453 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:09.458547 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:54:09 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:09.482641 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.458987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51449 (local address 127.0.97.62:38367)
0929 00:54:09.464834 (+ 5847us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:09.464839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:09.464855 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:09.464903 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:09.464906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:09.465001 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:09.465135 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:09.467768 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.468768 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.471190 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.471768 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.471916 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.476914 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:09.476920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:09.476923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:09.476953 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:09.479383 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:09.480056 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:09.480064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:09.480067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:09.480189 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:09.480621 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:09.480627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:09.480636 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:09.480895 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:09.481071 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:09.482209 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0929 00:54:09.482374 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5750,"thread_start_us":5667,"threads_started":1}
I20260929 00:54:09.484812 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.459546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.1:51449)
0929 00:54:09.460387 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.460436 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:09.463721 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:09.465460 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:09.465475 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:09.466457 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:09.467557 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.467590 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.469025 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.469030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.470976 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.470995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.472213 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.472222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.472593 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.473360 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:09.473384 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:09.476688 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:09.479518 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:09.479528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:09.479543 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:09.479940 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:09.480304 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:09.480309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:09.480313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:09.480483 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:09.481081 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:09.481094 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:09.481716 (+ 622us) client_negotiation.cc:769] Sending connection context
0929 00:54:09.482128 (+ 412us) client_negotiation.cc:240] Negotiation successful
0929 00:54:09.482813 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":362,"threads_started":1}
I20260929 00:54:09.509601 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.494667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.1:44833)
0929 00:54:09.495592 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.495617 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:09.495780 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:09.496323 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:09.496326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:09.497017 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:09.497363 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.497372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.498332 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.498340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.499351 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.499362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.500077 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.500092 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.500250 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.500930 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:09.500961 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:09.502063 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:09.506346 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:09.506349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:09.506352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:09.506734 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:09.507178 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:09.507185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:09.507191 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:09.507355 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:09.508058 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:09.508061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:09.508312 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 00:54:09.508492 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 00:54:09.508897 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":535,"threads_started":1}
I20260929 00:54:09.510169 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.494579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44833 (local address 127.0.97.62:38367)
0929 00:54:09.494796 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:09.494804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:09.495719 (+ 915us) server_negotiation.cc:407] Connection header received
0929 00:54:09.495935 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:09.495942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:09.496036 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:09.496156 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:09.497507 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.498209 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.499492 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.499886 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.500009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.502352 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:09.502359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:09.502366 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:09.502409 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:09.506225 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:09.506849 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:09.506854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:09.506858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:09.507018 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:09.507478 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:09.507485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:09.507488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:09.507861 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:09.508007 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:09.509582 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0929 00:54:09.509824 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 00:54:09.517215 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34801
I20260929 00:54:09.517251 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34801 every 8 connection(s)
I20260929 00:54:09.518371 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:09.520280 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38367
I20260929 00:54:09.520501 1718 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:09.521117 1718 heartbeater.cc:511] Master 127.0.97.62:38367 requested a full tablet report, sending...
I20260929 00:54:09.522972 1511 ts_manager.cc:194] Registered new tserver with Master: 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801)
I20260929 00:54:09.524128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1582
I20260929 00:54:09.524326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:09.524804 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51449
I20260929 00:54:09.525092 388 external_mini_cluster.cc:1428] Setting key 62b0e9da151f40d3e4725c3d2527c6a8
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:09.563719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38367
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:09.877810 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:09.878082 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:09.878221 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:09.885970 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:09.886083 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:09.886227 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:09.886351 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:09.896174 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:09.897439 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:09.898754 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:09.913108 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:09.913012 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:09.913515 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:09.913875 1722 server_base.cc:1034] running on GCE node
I20260929 00:54:09.914572 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:09.915469 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:09.916707 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790643249916637 us; error 40 us; skew 500 ppm
I20260929 00:54:09.916996 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:09 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:09.924733 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:09.926200 1722 webserver.cc:460] Webserver started at http://127.0.97.2:44457/ using document root <none> and password file <none>
I20260929 00:54:09.926878 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:09.927008 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:09.927330 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:09.930333 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "6deb98735585450bbad21c87d4f58645"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "f0aefa509b73b3aa9239bff6d1263758"
server_key_iv: "012b962954b9c00380c1226358dcc4b0"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.931123 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6deb98735585450bbad21c87d4f58645"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "f0aefa509b73b3aa9239bff6d1263758"
server_key_iv: "012b962954b9c00380c1226358dcc4b0"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.936038 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:54:09.939414 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:09.940738 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:09.940900 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "6deb98735585450bbad21c87d4f58645"
format_stamp: "Formatted at 2026-09-29 00:54:09 on dist-test-slave-pzsx"
server_key: "f0aefa509b73b3aa9239bff6d1263758"
server_key_iv: "012b962954b9c00380c1226358dcc4b0"
server_key_version: "encryptionkey@0"
I20260929 00:54:09.941123 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:09.972803 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:09.977352 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:09.977646 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:09.980253 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:09.980362 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:09.980548 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:09.980639 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 00:54:09 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:10.003757 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.981012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33503 (local address 127.0.97.62:38367)
0929 00:54:09.981210 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:09.981216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:09.982635 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 00:54:09.984688 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:09.984693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:09.984782 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:09.984889 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:09.987533 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.988934 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.991119 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.991889 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.992047 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.997533 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:09.997538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:09.997541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:09.997571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:09.999970 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.000947 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.000955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.000965 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.001140 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.001893 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.001900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.001904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.002298 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:10.002453 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.003183 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.003394 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:54:10.004964 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:09.981576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.2:33503)
0929 00:54:09.982443 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:09.982496 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:09.984339 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:09.985318 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:09.985332 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:09.986290 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:09.987263 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.987302 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.989083 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.989096 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.990976 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:09.990987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:09.992412 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:09.992426 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:09.992839 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:09.993767 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:09.993808 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:09.997304 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:10.000107 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.000117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.000133 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.000821 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.001269 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.001283 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.001288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.001730 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.002435 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:10.002443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:10.002791 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.003075 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.003629 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":372,"threads_started":1}
I20260929 00:54:10.027215 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.010565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49151 (local address 127.0.97.62:38367)
0929 00:54:10.010729 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.010733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.011405 (+ 672us) server_negotiation.cc:407] Connection header received
0929 00:54:10.011633 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.011638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.011707 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.011807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:10.013227 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.014094 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.016540 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.017123 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.017280 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.020231 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:10.020239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:10.020246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:10.020286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:10.023721 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.024859 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.024865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.024869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.025024 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.025509 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.025518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.025523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.026107 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:10.026317 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.026643 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.026851 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:54:10.027343 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.010687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.2:49151)
0929 00:54:10.011293 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.011314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.011482 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.011832 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.011836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.012572 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:10.013079 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.013105 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.014229 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.014234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.016372 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.016383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.017367 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.017377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.017643 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.018487 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:10.018518 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:10.020028 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:10.023853 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.023859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.023863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.024672 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.025151 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.025155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.025159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.025359 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.026252 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:10.026255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:10.026407 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.026555 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.026866 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260929 00:54:10.033599 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39507
I20260929 00:54:10.033660 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39507 every 8 connection(s)
I20260929 00:54:10.035326 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:10.037506 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38367
I20260929 00:54:10.037868 1857 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:10.038731 1857 heartbeater.cc:511] Master 127.0.97.62:38367 requested a full tablet report, sending...
I20260929 00:54:10.040194 1511 ts_manager.cc:194] Registered new tserver with Master: 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507)
I20260929 00:54:10.040910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1722
I20260929 00:54:10.041098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:10.041329 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33503
I20260929 00:54:10.041616 388 external_mini_cluster.cc:1428] Setting key da84d07ab1599980b81395dcfb0c1d72
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:10.079159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38367
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:10.397446 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:10.397723 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:10.397872 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:10.405242 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:10.405344 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:10.405512 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:10.405627 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:10.415544 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:10.416968 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:10.418285 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:10.431121 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:10.431121 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:10.431135 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:10.432205 1861 server_base.cc:1034] running on GCE node
I20260929 00:54:10.432866 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:10.433765 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:10.435007 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790643250434940 us; error 53 us; skew 500 ppm
I20260929 00:54:10.435308 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:10 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:10.438503 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:10.439917 1861 webserver.cc:460] Webserver started at http://127.0.97.3:37971/ using document root <none> and password file <none>
I20260929 00:54:10.440682 1861 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:10.440809 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:10.441131 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:10.443950 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "80298f953c834640a44107008b74dbe9"
format_stamp: "Formatted at 2026-09-29 00:54:10 on dist-test-slave-pzsx"
server_key: "5a4573bec0493a6dd471e7981dfb30f2"
server_key_iv: "4a1bdd5c878723b4d488b5d8870c15a5"
server_key_version: "encryptionkey@0"
I20260929 00:54:10.444912 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "80298f953c834640a44107008b74dbe9"
format_stamp: "Formatted at 2026-09-29 00:54:10 on dist-test-slave-pzsx"
server_key: "5a4573bec0493a6dd471e7981dfb30f2"
server_key_iv: "4a1bdd5c878723b4d488b5d8870c15a5"
server_key_version: "encryptionkey@0"
I20260929 00:54:10.450241 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:10.453826 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:10.455266 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:10.455440 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "80298f953c834640a44107008b74dbe9"
format_stamp: "Formatted at 2026-09-29 00:54:10 on dist-test-slave-pzsx"
server_key: "5a4573bec0493a6dd471e7981dfb30f2"
server_key_iv: "4a1bdd5c878723b4d488b5d8870c15a5"
server_key_version: "encryptionkey@0"
I20260929 00:54:10.455680 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:10.471149 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:10.475019 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:10.475294 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:10.477998 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:10.478116 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:10.478261 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:10.478335 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:54:10 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:10.502475 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.478361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37347 (local address 127.0.97.62:38367)
0929 00:54:10.478541 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.478547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.480032 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 00:54:10.482099 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.482109 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.482215 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.482332 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:10.484702 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.485853 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.489283 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.489835 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.489970 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.495140 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:10.495145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:10.495148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:10.495179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:10.498673 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.499939 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.499948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.499955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.500138 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.500738 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.500744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.500747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.501032 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:10.501188 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.501900 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.502124 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:54:10.503543 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.478997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.3:37347)
0929 00:54:10.479838 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.479890 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.481810 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.482598 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.482610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.483350 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:10.484430 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.484460 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.486057 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.486076 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.489087 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.489127 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.490293 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.490303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.490667 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.491534 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:10.491558 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:10.494900 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:10.498865 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.498880 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.498902 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.499755 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.500299 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.500306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.500315 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.500608 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.501181 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:10.501189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:10.501529 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.501799 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.502304 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":374,"threads_started":1}
I20260929 00:54:10.527910 1718 heartbeater.cc:503] Master 127.0.97.62:38367 was elected leader, sending a full tablet report...
I20260929 00:54:10.531476 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36109
I20260929 00:54:10.531493 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36109 every 8 connection(s)
I20260929 00:54:10.532006 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.515298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.97.3:37321)
0929 00:54:10.516106 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.516133 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.516336 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.516929 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.516937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.517540 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:10.518068 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.518085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.519514 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.519518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.521298 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.521312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.522569 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.522576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.522733 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.523485 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:10.523501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:10.524159 (+ 658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:10.528734 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.528739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.528744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.529475 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.529913 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:10.529917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:10.529919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:10.530035 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:10.530699 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:10.530707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:10.530939 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.531115 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.531534 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":405,"threads_started":1}
I20260929 00:54:10.533016 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.515321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37321 (local address 127.0.97.62:38367)
0929 00:54:10.515550 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.515558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.516268 (+ 710us) server_negotiation.cc:407] Connection header received
0929 00:54:10.516538 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.516543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.516639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.516741 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:10.518256 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.519324 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.521458 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.522354 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.522519 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.524360 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:10.524369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:10.524377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:10.524428 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:10.528546 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.529652 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.529657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.529660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.529804 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.530144 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.530150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.530153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.530523 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:10.532273 (+ 1750us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.532417 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.532677 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:54:10.533149 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:10.535102 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38367
I20260929 00:54:10.535327 1996 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:10.536089 1996 heartbeater.cc:511] Master 127.0.97.62:38367 requested a full tablet report, sending...
I20260929 00:54:10.536911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 1861
I20260929 00:54:10.537110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:10.537606 1511 ts_manager.cc:194] Registered new tserver with Master: 80298f953c834640a44107008b74dbe9 (127.0.97.3:36109)
I20260929 00:54:10.537719 388 external_mini_cluster.cc:1428] Setting key 706f5994ea631047fe5bcdb237d11ad8
I20260929 00:54:10.538522 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37347
I20260929 00:54:10.540701 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:54:10 dist-test-slave-pzsx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:54:10 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:10.583622 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.570831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.0.97.62:38367)
0929 00:54:10.571002 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.571006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.571322 (+ 316us) server_negotiation.cc:407] Connection header received
0929 00:54:10.571463 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.571467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.571528 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.571628 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:10.572987 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.573707 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.575116 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.575527 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.575625 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.579199 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:10.579204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:10.579208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:10.579239 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:10.581624 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.582156 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.582159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.582161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.582267 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:10.582571 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:10.582574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:10.582576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:10.582817 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:10.582983 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.583198 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.583351 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:54:10.587383 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:10.593986 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:10.629609 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.605300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.0.97.2:39507)
0929 00:54:10.620409 (+ 15109us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.620422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.621296 (+ 874us) server_negotiation.cc:407] Connection header received
0929 00:54:10.621427 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.621437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.621725 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.621946 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:10.622888 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.624327 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.627082 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.628468 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.628616 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.628832 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.629150 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.629318 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14938,"thread_start_us":14806,"threads_started":1}
I20260929 00:54:10.629647 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.605249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39507 (local address 127.0.0.1:43626)
0929 00:54:10.620900 (+ 15651us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.620947 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.621153 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.622196 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.622202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.622248 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:10.622683 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.622698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.624452 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.624460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.626877 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.626894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.628689 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.628695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.628896 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.628956 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.629097 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.629269 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15411,"spinlock_wait_cycles":9344,"thread_start_us":15241,"threads_started":1}
I20260929 00:54:10.631469 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.607490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34801 (local address 127.0.0.1:38040)
0929 00:54:10.623154 (+ 15664us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.623172 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.623284 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.624117 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.624120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.624166 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:10.624599 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.624615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.626433 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.626442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.628712 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.628725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.630749 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.630760 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.630904 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.630941 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.631006 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.631119 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15534,"mutex_wait_us":86,"thread_start_us":14347,"threads_started":1}
I20260929 00:54:10.631521 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.608208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.97.1:34801)
0929 00:54:10.621700 (+ 13492us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.621708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.623398 (+ 1690us) server_negotiation.cc:407] Connection header received
0929 00:54:10.623489 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.623495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.623716 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.623887 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:10.624806 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.626269 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.628905 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.630550 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.630691 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.630878 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.631093 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.631250 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13345,"thread_start_us":13219,"threads_started":1}
I20260929 00:54:10.632160 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.605861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.0.97.3:36109)
0929 00:54:10.622879 (+ 17018us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.622886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.622919 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:54:10.623007 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.623014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.623241 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.623417 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:10.624257 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.625873 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.628338 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.630339 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.631444 (+ 1105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.631708 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.631779 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.631941 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16836,"thread_start_us":16637,"threads_started":1}
I20260929 00:54:10.632411 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.605320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36109 (local address 127.0.0.1:50704)
0929 00:54:10.622132 (+ 16812us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.622150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.622259 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.623664 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.623669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.623692 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:10.624043 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.624052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.626285 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.626290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.628184 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.628197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.630609 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.630614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.630827 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.630858 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.630936 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.631073 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16662,"spinlock_wait_cycles":6784,"thread_start_us":17603,"threads_started":1}
I20260929 00:54:10.633885 1787 tablet_service.cc:1467] Processing CreateTablet for tablet cae1b852a3d743328d2ec7e61113be01 (DEFAULT_TABLE table=test-table [id=60706d0dcc4b4af4907829ccaa6c410c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:10.635356 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae1b852a3d743328d2ec7e61113be01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:10.634960 1648 tablet_service.cc:1467] Processing CreateTablet for tablet cae1b852a3d743328d2ec7e61113be01 (DEFAULT_TABLE table=test-table [id=60706d0dcc4b4af4907829ccaa6c410c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:10.635869 1926 tablet_service.cc:1467] Processing CreateTablet for tablet cae1b852a3d743328d2ec7e61113be01 (DEFAULT_TABLE table=test-table [id=60706d0dcc4b4af4907829ccaa6c410c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:10.636222 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae1b852a3d743328d2ec7e61113be01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:10.636971 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae1b852a3d743328d2ec7e61113be01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:10.650074 2017 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Bootstrap starting.
I20260929 00:54:10.651505 2015 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Bootstrap starting.
I20260929 00:54:10.652774 2016 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Bootstrap starting.
I20260929 00:54:10.654022 2017 tablet_bootstrap.cc:654] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:10.655079 2015 tablet_bootstrap.cc:654] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:10.655361 2017 log.cc:824] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:10.656123 2016 tablet_bootstrap.cc:654] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:10.656239 2015 log.cc:824] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:10.657261 2016 log.cc:824] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:10.658479 2017 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: No bootstrap required, opened a new log
I20260929 00:54:10.658744 2017 ts_tablet_manager.cc:1397] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 00:54:10.659471 2015 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: No bootstrap required, opened a new log
I20260929 00:54:10.659720 2015 ts_tablet_manager.cc:1397] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 00:54:10.660596 2016 tablet_bootstrap.cc:492] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: No bootstrap required, opened a new log
I20260929 00:54:10.660828 2016 ts_tablet_manager.cc:1397] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 00:54:10.667397 2015 raft_consensus.cc:348] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.667829 2015 raft_consensus.cc:374] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:10.667932 2015 raft_consensus.cc:729] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80298f953c834640a44107008b74dbe9, State: Initialized, Role: FOLLOWER
I20260929 00:54:10.668633 2015 consensus_queue.cc:260] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.669884 1996 heartbeater.cc:503] Master 127.0.97.62:38367 was elected leader, sending a full tablet report...
I20260929 00:54:10.669718 2017 raft_consensus.cc:348] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.670125 2017 raft_consensus.cc:374] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:10.670224 2017 raft_consensus.cc:729] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb98735585450bbad21c87d4f58645, State: Initialized, Role: FOLLOWER
I20260929 00:54:10.670509 2015 ts_tablet_manager.cc:1428] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 00:54:10.670770 2017 consensus_queue.cc:260] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.671782 1857 heartbeater.cc:503] Master 127.0.97.62:38367 was elected leader, sending a full tablet report...
I20260929 00:54:10.672677 2017 ts_tablet_manager.cc:1428] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260929 00:54:10.673161 2016 raft_consensus.cc:348] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.673561 2016 raft_consensus.cc:374] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:10.673664 2016 raft_consensus.cc:729] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df63b96cb5a41af9966a81a589cd065, State: Initialized, Role: FOLLOWER
I20260929 00:54:10.674302 2016 consensus_queue.cc:260] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.675854 2016 ts_tablet_manager.cc:1428] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260929 00:54:10.770666 1719 tablet.cc:2369] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:10.784921 1997 tablet.cc:2369] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:10.787374 1858 tablet.cc:2369] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:10.888463 2021 raft_consensus.cc:482] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:10.888809 2021 raft_consensus.cc:504] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.890548 2021 leader_election.cc:288] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507), 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801)
I20260929 00:54:10.897351 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.890903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42917 (local address 127.0.97.2:39507)
0929 00:54:10.891106 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.891114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.891395 (+ 281us) server_negotiation.cc:407] Connection header received
0929 00:54:10.891661 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.891666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.891747 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.891850 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:10.892377 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.893341 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.895051 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.896012 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.896115 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.896194 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.896971 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.897062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:54:10.897375 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.890976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39507 (local address 127.0.97.3:42917)
0929 00:54:10.891280 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.891305 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.891466 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.891880 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.891883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.891904 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:10.892209 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.892221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.893467 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.893472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.894888 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.894899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.896613 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.896619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.896812 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.896848 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.896949 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.897081 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"spinlock_wait_cycles":140672}
I20260929 00:54:10.897976 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.891035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59129 (local address 127.0.97.1:34801)
0929 00:54:10.891216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.891223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.891998 (+ 775us) server_negotiation.cc:407] Connection header received
0929 00:54:10.892183 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.892188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.892278 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.892369 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:10.892975 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.893811 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.895620 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.897028 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.897152 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.897252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.897616 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.897750 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:54:10.898494 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.890984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34801 (local address 127.0.97.3:59129)
0929 00:54:10.891880 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:10.891899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:10.892044 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:10.892394 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:10.892398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:10.892422 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:10.892812 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.892823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.893941 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.893945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.895470 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:10.895483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.897232 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.897238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:10.897425 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.897460 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:54:10.898158 (+ 698us) client_negotiation.cc:240] Negotiation successful
0929 00:54:10.898259 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":35,"thread_start_us":520,"threads_started":1}
I20260929 00:54:10.898362 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae1b852a3d743328d2ec7e61113be01" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb98735585450bbad21c87d4f58645" is_pre_election: true
I20260929 00:54:10.898854 1812 raft_consensus.cc:2454] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 0.
I20260929 00:54:10.899494 1879 leader_election.cc:302] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb98735585450bbad21c87d4f58645, 80298f953c834640a44107008b74dbe9; no voters:
I20260929 00:54:10.899438 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae1b852a3d743328d2ec7e61113be01" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065" is_pre_election: true
I20260929 00:54:10.899855 2021 raft_consensus.cc:2781] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:10.899889 1673 raft_consensus.cc:2454] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 0.
I20260929 00:54:10.900012 2021 raft_consensus.cc:482] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:10.900127 2021 raft_consensus.cc:3037] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:10.902370 2021 raft_consensus.cc:504] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.902941 2021 leader_election.cc:288] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 election: Requested vote from peers 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507), 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801)
I20260929 00:54:10.903332 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae1b852a3d743328d2ec7e61113be01" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb98735585450bbad21c87d4f58645"
I20260929 00:54:10.903379 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae1b852a3d743328d2ec7e61113be01" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065"
I20260929 00:54:10.903549 1812 raft_consensus.cc:3037] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:10.903549 1673 raft_consensus.cc:3037] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:10.906741 1812 raft_consensus.cc:2454] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 1.
I20260929 00:54:10.906733 1673 raft_consensus.cc:2454] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 1.
I20260929 00:54:10.907143 1880 leader_election.cc:302] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df63b96cb5a41af9966a81a589cd065, 80298f953c834640a44107008b74dbe9; no voters:
I20260929 00:54:10.907459 2021 raft_consensus.cc:2781] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:10.907899 2021 raft_consensus.cc:686] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 1 LEADER]: Becoming Leader. State: Replica: 80298f953c834640a44107008b74dbe9, State: Running, Role: LEADER
I20260929 00:54:10.908371 2021 consensus_queue.cc:237] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } }
I20260929 00:54:10.912810 1509 catalog_manager.cc:5495] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80298f953c834640a44107008b74dbe9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "80298f953c834640a44107008b74dbe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:10.934114 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:10.928956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.0.97.3:36109)
0929 00:54:10.929122 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:10.929126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:10.929168 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:54:10.929436 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:10.929439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:10.929518 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:10.929640 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:10.929948 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.930635 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.931878 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:10.932259 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:10.932345 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:10.932796 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:10.933671 (+ 875us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:10.933774 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:10.933862 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 00:54:10.933921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:54:10.941679 1812 raft_consensus.cc:1261] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Refusing update from remote peer 80298f953c834640a44107008b74dbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:10.941679 1673 raft_consensus.cc:1261] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Refusing update from remote peer 80298f953c834640a44107008b74dbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:10.942726 2025 consensus_queue.cc:1035] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:10.943082 2021 consensus_queue.cc:1035] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:10.956890 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334474755847413760 to 7334474755724558336. Current Snapshot: MvccSnapshot[applied={T|T < 7334474755847413760}]
I20260929 00:54:10.975677 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:10.975976 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:10.979519 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Sending DeleteTablet for 3 replicas of tablet cae1b852a3d743328d2ec7e61113be01
I20260929 00:54:10.980633 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet cae1b852a3d743328d2ec7e61113be01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50704
I20260929 00:54:10.980649 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet cae1b852a3d743328d2ec7e61113be01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43626
I20260929 00:54:10.980849 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet cae1b852a3d743328d2ec7e61113be01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38040
I20260929 00:54:10.981702 2039 tablet_replica.cc:330] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: stopping tablet replica
I20260929 00:54:10.982295 2039 raft_consensus.cc:2229] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:10.982931 2039 raft_consensus.cc:2258] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:10.982738 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:10.983573 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:10.984149 2038 tablet_replica.cc:330] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: stopping tablet replica
I20260929 00:54:10.984601 2038 raft_consensus.cc:2229] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:10.985677 2038 raft_consensus.cc:2258] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:10.987023 2039 ts_tablet_manager.cc:1905] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:10.987802 2038 ts_tablet_manager.cc:1905] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:10.992250 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (DEFAULT_TABLE table=test-table [id=31009f20f4654422977417776e181bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:10.992863 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:10.995970 2039 ts_tablet_manager.cc:1918] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:10.996136 2039 log.cc:1196] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/cae1b852a3d743328d2ec7e61113be01
I20260929 00:54:10.996681 2039 ts_tablet_manager.cc:1939] T cae1b852a3d743328d2ec7e61113be01 P 6deb98735585450bbad21c87d4f58645: Deleting consensus metadata
I20260929 00:54:10.996659 2038 ts_tablet_manager.cc:1918] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:10.997005 2038 log.cc:1196] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/cae1b852a3d743328d2ec7e61113be01
I20260929 00:54:10.997670 2038 ts_tablet_manager.cc:1939] T cae1b852a3d743328d2ec7e61113be01 P 80298f953c834640a44107008b74dbe9: Deleting consensus metadata
I20260929 00:54:10.998543 1497 catalog_manager.cc:4841] TS 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507): tablet cae1b852a3d743328d2ec7e61113be01 (table test-table [id=60706d0dcc4b4af4907829ccaa6c410c]) successfully deleted
I20260929 00:54:10.999521 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (DEFAULT_TABLE table=test-table [id=31009f20f4654422977417776e181bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:10.999914 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:11.000141 2016 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Bootstrap starting.
I20260929 00:54:11.001331 2040 tablet_replica.cc:330] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: stopping tablet replica
I20260929 00:54:11.001823 2016 tablet_bootstrap.cc:654] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.001859 2040 raft_consensus.cc:2229] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:11.002470 2040 raft_consensus.cc:2258] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:11.004441 2040 ts_tablet_manager.cc:1905] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:11.006776 1495 catalog_manager.cc:4841] TS 80298f953c834640a44107008b74dbe9 (127.0.97.3:36109): tablet cae1b852a3d743328d2ec7e61113be01 (table test-table [id=60706d0dcc4b4af4907829ccaa6c410c]) successfully deleted
I20260929 00:54:11.007272 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (DEFAULT_TABLE table=test-table [id=31009f20f4654422977417776e181bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:11.008158 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:11.008484 2017 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Bootstrap starting.
I20260929 00:54:11.010123 2017 tablet_bootstrap.cc:654] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.011619 2040 ts_tablet_manager.cc:1918] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:11.011825 2040 log.cc:1196] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/cae1b852a3d743328d2ec7e61113be01
I20260929 00:54:11.012389 2040 ts_tablet_manager.cc:1939] T cae1b852a3d743328d2ec7e61113be01 P 6df63b96cb5a41af9966a81a589cd065: Deleting consensus metadata
I20260929 00:54:11.014358 1498 catalog_manager.cc:4841] TS 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801): tablet cae1b852a3d743328d2ec7e61113be01 (table test-table [id=60706d0dcc4b4af4907829ccaa6c410c]) successfully deleted
I20260929 00:54:11.016729 2015 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Bootstrap starting.
I20260929 00:54:11.018448 2015 tablet_bootstrap.cc:654] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.024113 2017 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: No bootstrap required, opened a new log
I20260929 00:54:11.024276 2017 ts_tablet_manager.cc:1397] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260929 00:54:11.025142 2017 raft_consensus.cc:348] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.025367 2017 raft_consensus.cc:374] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.025466 2017 raft_consensus.cc:729] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb98735585450bbad21c87d4f58645, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.025635 2015 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: No bootstrap required, opened a new log
I20260929 00:54:11.025784 2015 ts_tablet_manager.cc:1397] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 00:54:11.025725 2017 consensus_queue.cc:260] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.026340 2017 ts_tablet_manager.cc:1428] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:54:11.027197 2016 tablet_bootstrap.cc:492] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: No bootstrap required, opened a new log
I20260929 00:54:11.027346 2016 ts_tablet_manager.cc:1397] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20260929 00:54:11.028158 2015 raft_consensus.cc:348] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.028139 2016 raft_consensus.cc:348] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.028357 2016 raft_consensus.cc:374] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.028365 2015 raft_consensus.cc:374] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.028445 2016 raft_consensus.cc:729] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df63b96cb5a41af9966a81a589cd065, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.028460 2015 raft_consensus.cc:729] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80298f953c834640a44107008b74dbe9, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.028757 2016 consensus_queue.cc:260] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.030473 2016 ts_tablet_manager.cc:1428] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:11.028762 2015 consensus_queue.cc:260] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.031975 2015 ts_tablet_manager.cc:1428] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260929 00:54:11.100250 2033 raft_consensus.cc:482] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:11.100605 2033 raft_consensus.cc:504] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.101353 2033 leader_election.cc:288] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801), 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507)
I20260929 00:54:11.101897 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc2f04e67c47d5b40ab2ed99f0cc26" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065" is_pre_election: true
I20260929 00:54:11.102007 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc2f04e67c47d5b40ab2ed99f0cc26" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb98735585450bbad21c87d4f58645" is_pre_election: true
I20260929 00:54:11.102162 1673 raft_consensus.cc:2454] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 0.
I20260929 00:54:11.102221 1812 raft_consensus.cc:2454] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 0.
I20260929 00:54:11.102551 1880 leader_election.cc:302] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df63b96cb5a41af9966a81a589cd065, 80298f953c834640a44107008b74dbe9; no voters:
I20260929 00:54:11.102883 2033 raft_consensus.cc:2781] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:11.103032 2033 raft_consensus.cc:482] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:11.103139 2033 raft_consensus.cc:3037] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.106554 2033 raft_consensus.cc:504] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.107223 2033 leader_election.cc:288] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 election: Requested vote from peers 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801), 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507)
I20260929 00:54:11.107620 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc2f04e67c47d5b40ab2ed99f0cc26" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065"
I20260929 00:54:11.107683 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc2f04e67c47d5b40ab2ed99f0cc26" candidate_uuid: "80298f953c834640a44107008b74dbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb98735585450bbad21c87d4f58645"
I20260929 00:54:11.107817 1673 raft_consensus.cc:3037] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.107889 1812 raft_consensus.cc:3037] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.109875 1812 raft_consensus.cc:2454] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 1.
I20260929 00:54:11.110232 1879 leader_election.cc:302] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb98735585450bbad21c87d4f58645, 80298f953c834640a44107008b74dbe9; no voters:
I20260929 00:54:11.110575 2033 raft_consensus.cc:2781] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:11.111073 2033 raft_consensus.cc:686] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 1 LEADER]: Becoming Leader. State: Replica: 80298f953c834640a44107008b74dbe9, State: Running, Role: LEADER
I20260929 00:54:11.111366 2033 consensus_queue.cc:237] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.111476 1673 raft_consensus.cc:2454] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80298f953c834640a44107008b74dbe9 in term 1.
I20260929 00:54:11.114113 1510 catalog_manager.cc:5495] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80298f953c834640a44107008b74dbe9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "80298f953c834640a44107008b74dbe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } health_report { overall_health: HEALTHY } } }
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:11.185829 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.170968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38367 (local address 127.0.0.1:40872)
0929 00:54:11.171593 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.171615 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.171780 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.172135 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.172142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.172870 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:11.173098 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.173107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.174140 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.174148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.175137 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.175153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.176310 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.176315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.176424 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.177176 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:11.177192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:11.179498 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:11.183519 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:11.183528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:11.183534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:11.183901 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:11.184292 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:11.184295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:11.184297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:11.184419 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:11.185063 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:11.185069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:11.185210 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.185340 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.185545 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260929 00:54:11.186218 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.171008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.97.62:38367)
0929 00:54:11.171862 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.171867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.171891 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:54:11.171941 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.171943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.171993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.172100 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.173287 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.174012 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.175312 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.176135 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.176291 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.179695 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.179702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.179708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.179748 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.183377 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.184018 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.184024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.184028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.184177 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.184555 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.184560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.184564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.184940 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.185097 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.185476 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.185711 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"spinlock_wait_cycles":8960,"thread_start_us":591,"threads_started":1}
I20260929 00:54:11.191103 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:54:11.192107 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:11.198573 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 084d12e417a54c9a84cf5ae8784e13bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5cc8f2a23708495ea087d33408fd7811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:11.198787 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 084d12e417a54c9a84cf5ae8784e13bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5cc8f2a23708495ea087d33408fd7811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:11.198927 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 084d12e417a54c9a84cf5ae8784e13bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5cc8f2a23708495ea087d33408fd7811]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:11.199137 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084d12e417a54c9a84cf5ae8784e13bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:11.199314 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084d12e417a54c9a84cf5ae8784e13bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:11.199412 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084d12e417a54c9a84cf5ae8784e13bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:11.207185 2015 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: Bootstrap starting.
I20260929 00:54:11.207233 2017 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: Bootstrap starting.
I20260929 00:54:11.207278 2016 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: Bootstrap starting.
I20260929 00:54:11.208930 2017 tablet_bootstrap.cc:654] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.208954 2015 tablet_bootstrap.cc:654] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.209033 2016 tablet_bootstrap.cc:654] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:11.211864 2016 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: No bootstrap required, opened a new log
I20260929 00:54:11.212049 2016 ts_tablet_manager.cc:1397] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 00:54:11.212817 2016 raft_consensus.cc:348] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.213065 2016 raft_consensus.cc:374] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.213143 2016 raft_consensus.cc:729] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df63b96cb5a41af9966a81a589cd065, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.213501 2016 consensus_queue.cc:260] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.213937 2017 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: No bootstrap required, opened a new log
I20260929 00:54:11.214083 2017 ts_tablet_manager.cc:1397] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 00:54:11.213948 2023 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.214300 2023 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.214757 2017 raft_consensus.cc:348] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.214946 2017 raft_consensus.cc:374] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.215010 2017 raft_consensus.cc:729] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb98735585450bbad21c87d4f58645, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.215233 2016 ts_tablet_manager.cc:1428] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 00:54:11.215229 2017 consensus_queue.cc:260] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.215775 2017 ts_tablet_manager.cc:1428] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 00:54:11.215700 2022 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.216065 2022 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.217283 2015 tablet_bootstrap.cc:492] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: No bootstrap required, opened a new log
I20260929 00:54:11.217458 2015 ts_tablet_manager.cc:1397] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260929 00:54:11.218259 2015 raft_consensus.cc:348] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.218500 2015 raft_consensus.cc:374] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:11.218590 2015 raft_consensus.cc:729] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80298f953c834640a44107008b74dbe9, State: Initialized, Role: FOLLOWER
I20260929 00:54:11.218845 2015 consensus_queue.cc:260] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.219293 2033 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.219473 2015 ts_tablet_manager.cc:1428] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:11.219614 2033 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.297760 2022 raft_consensus.cc:482] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:11.298026 2022 raft_consensus.cc:504] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.300173 2022 leader_election.cc:288] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801), 80298f953c834640a44107008b74dbe9 (127.0.97.3:36109)
I20260929 00:54:11.308817 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.301031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56655 (local address 127.0.97.1:34801)
0929 00:54:11.301233 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.301240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.302274 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 00:54:11.302412 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.302418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.302537 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.302642 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:11.303186 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.303894 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.306158 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.307635 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.307773 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.307883 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.308350 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.308483 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:54:11.309505 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.300551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34801 (local address 127.0.97.2:56655)
0929 00:54:11.302151 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.302167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.302286 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.302669 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.302672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.302690 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:11.303001 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.303008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.304019 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.304027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.305982 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.305997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.307954 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.307961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.308159 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.308195 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.308854 (+ 659us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.309119 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"thread_start_us":1278,"threads_started":1}
I20260929 00:54:11.309952 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.300390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36109 (local address 127.0.97.2:38705)
0929 00:54:11.301738 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.301760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.302005 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.302822 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.302828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.302857 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:11.303332 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.303344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.304994 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.304999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.307043 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.307059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.309254 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.309258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.309382 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.309405 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.309518 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.309650 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":1015,"threads_started":1}
I20260929 00:54:11.310158 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084d12e417a54c9a84cf5ae8784e13bd" candidate_uuid: "6deb98735585450bbad21c87d4f58645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065" is_pre_election: true
I20260929 00:54:11.310490 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.301425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38705 (local address 127.0.97.3:36109)
0929 00:54:11.301602 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.301610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.301858 (+ 248us) server_negotiation.cc:407] Connection header received
0929 00:54:11.302190 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.302198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.302283 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.302370 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:11.303527 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.304851 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.307241 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.308840 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.308978 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.309097 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.309539 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.309950 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:54:11.310475 1673 raft_consensus.cc:2454] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb98735585450bbad21c87d4f58645 in term 0.
I20260929 00:54:11.310961 1741 leader_election.cc:302] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb98735585450bbad21c87d4f58645, 6df63b96cb5a41af9966a81a589cd065; no voters:
I20260929 00:54:11.311357 2022 raft_consensus.cc:2781] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:11.311488 2022 raft_consensus.cc:482] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:11.311347 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084d12e417a54c9a84cf5ae8784e13bd" candidate_uuid: "6deb98735585450bbad21c87d4f58645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80298f953c834640a44107008b74dbe9" is_pre_election: true
I20260929 00:54:11.311579 2022 raft_consensus.cc:3037] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.311800 1946 raft_consensus.cc:2454] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb98735585450bbad21c87d4f58645 in term 0.
I20260929 00:54:11.315431 2022 raft_consensus.cc:504] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.316176 2022 leader_election.cc:288] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [CANDIDATE]: Term 1 election: Requested vote from peers 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801), 80298f953c834640a44107008b74dbe9 (127.0.97.3:36109)
I20260929 00:54:11.316622 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084d12e417a54c9a84cf5ae8784e13bd" candidate_uuid: "6deb98735585450bbad21c87d4f58645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df63b96cb5a41af9966a81a589cd065"
I20260929 00:54:11.316708 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084d12e417a54c9a84cf5ae8784e13bd" candidate_uuid: "6deb98735585450bbad21c87d4f58645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80298f953c834640a44107008b74dbe9"
I20260929 00:54:11.316879 1673 raft_consensus.cc:3037] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.316921 1946 raft_consensus.cc:3037] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:11.319651 1946 raft_consensus.cc:2454] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb98735585450bbad21c87d4f58645 in term 1.
I20260929 00:54:11.320055 1738 leader_election.cc:302] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb98735585450bbad21c87d4f58645, 80298f953c834640a44107008b74dbe9; no voters:
I20260929 00:54:11.320320 2022 raft_consensus.cc:2781] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:11.320487 1673 raft_consensus.cc:2454] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb98735585450bbad21c87d4f58645 in term 1.
I20260929 00:54:11.321498 2022 raft_consensus.cc:686] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb98735585450bbad21c87d4f58645, State: Running, Role: LEADER
I20260929 00:54:11.321908 2022 consensus_queue.cc:237] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } }
I20260929 00:54:11.323848 2057 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: TxnStatusTablet state changed. Reason: New leader 6deb98735585450bbad21c87d4f58645. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.324249 2057 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:11.325016 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:54:11.325263 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:54:11.326706 1509 catalog_manager.cc:5495] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb98735585450bbad21c87d4f58645 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:11.348150 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.340930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.0.97.2:39507)
0929 00:54:11.341083 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.341096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.341116 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:54:11.341258 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.341263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.341379 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.341517 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:11.341980 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.343189 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.345353 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.346078 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.346219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.346775 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.347453 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.347601 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.347731 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.347809 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:54:11.348207 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.340481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39507 (local address 127.0.0.1:43638)
0929 00:54:11.340891 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.340914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.341064 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.341565 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.341573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.341591 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:11.341807 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.341819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.343336 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.343342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.345204 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.345220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.346261 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.346268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.346496 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.346576 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.347578 (+ 1002us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.347600 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.347713 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.347872 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260929 00:54:11.352958 1946 raft_consensus.cc:1261] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Refusing update from remote peer 6deb98735585450bbad21c87d4f58645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:11.353760 1673 raft_consensus.cc:1261] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Refusing update from remote peer 6deb98735585450bbad21c87d4f58645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:11.353806 2022 consensus_queue.cc:1035] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:11.354525 2057 consensus_queue.cc:1035] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:11.360116 2023 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: TxnStatusTablet state changed. Reason: New leader 6deb98735585450bbad21c87d4f58645. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.360455 2023 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.362721 2022 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.362982 2022 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:11.363158 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334474757533835264 to 7334474757417938944. Current Snapshot: MvccSnapshot[applied={T|T < 7334474757533835264}]
I20260929 00:54:11.365538 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334474757533835264 to 7334474757417938944. Current Snapshot: MvccSnapshot[applied={T|T < 7334474757533835264}]
I20260929 00:54:11.366411 2023 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.366650 2023 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6df63b96cb5a41af9966a81a589cd065: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.372049 2033 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: TxnStatusTablet state changed. Reason: New leader 6deb98735585450bbad21c87d4f58645. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.372419 2033 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.376473 2057 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.376787 2057 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 6deb98735585450bbad21c87d4f58645: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:11.380086 2033 tablet_replica.cc:439] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6deb98735585450bbad21c87d4f58645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 } } peers { permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 } } peers { permanent_uuid: "80298f953c834640a44107008b74dbe9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36109 } } }
I20260929 00:54:11.380393 2033 tablet_replica.cc:442] T 084d12e417a54c9a84cf5ae8784e13bd P 80298f953c834640a44107008b74dbe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:11.406188 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.398845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47731 (local address 127.0.97.2:39507)
0929 00:54:11.399025 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.399032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.399589 (+ 557us) server_negotiation.cc:407] Connection header received
0929 00:54:11.399809 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.399816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.399920 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.400051 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:11.400481 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.401806 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.403808 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.404610 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.404763 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.405284 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.405426 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.405519 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.405759 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.405852 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:54:11.406314 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.398694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39507 (local address 127.0.97.3:47731)
0929 00:54:11.399456 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.399478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.399635 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.400080 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.400086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.400106 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:11.400322 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.400334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.401956 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.401961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.403650 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.403665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.404779 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.404784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.404969 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.405041 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.405600 (+ 559us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.405610 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.405701 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.405856 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":489,"threads_started":1}
I20260929 00:54:11.417578 1812 raft_consensus.cc:1261] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Refusing update from remote peer 80298f953c834640a44107008b74dbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:11.418329 2025 consensus_queue.cc:1035] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb98735585450bbad21c87d4f58645" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:11.419371 1673 raft_consensus.cc:1261] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Refusing update from remote peer 80298f953c834640a44107008b74dbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:11.420112 2025 consensus_queue.cc:1035] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6df63b96cb5a41af9966a81a589cd065" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:11.483523 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.474594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47295 (local address 127.0.97.3:36109)
0929 00:54:11.474769 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.474776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.476182 (+ 1406us) server_negotiation.cc:407] Connection header received
0929 00:54:11.476398 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.476403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.476549 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.476649 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:11.477356 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.478526 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.480863 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.481654 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.481804 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.482325 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.482484 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.482590 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.483125 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.483211 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:54:11.484570 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.474924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36109 (local address 127.0.97.2:47295)
0929 00:54:11.476058 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:11.476082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:11.476237 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:11.476690 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:11.476697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:11.476712 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:11.477159 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.477174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.478683 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.478689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.480710 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:11.480730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.481856 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.481863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:11.482043 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.482130 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:11.482958 (+ 828us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:11.482968 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:54:11.483950 (+ 982us) client_negotiation.cc:240] Negotiation successful
0929 00:54:11.484133 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":1018,"threads_started":1}
I20260929 00:54:11.547588 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:11.547870 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:11.551692 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c25fbc6cba943f48a4654af2385622e: Sending DeleteTablet for 3 replicas of tablet 57cc2f04e67c47d5b40ab2ed99f0cc26
I20260929 00:54:11.552824 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38040
I20260929 00:54:11.553277 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43626
I20260929 00:54:11.554157 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50704
I20260929 00:54:11.558040 2073 tablet_replica.cc:330] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: stopping tablet replica
I20260929 00:54:11.558288 2073 raft_consensus.cc:2229] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:11.558579 2073 raft_consensus.cc:2258] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:11.559680 2073 ts_tablet_manager.cc:1905] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:11.566964 2073 ts_tablet_manager.cc:1918] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:11.567189 2073 log.cc:1196] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/57cc2f04e67c47d5b40ab2ed99f0cc26
I20260929 00:54:11.567723 2073 ts_tablet_manager.cc:1939] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6df63b96cb5a41af9966a81a589cd065: Deleting consensus metadata
I20260929 00:54:11.568948 1498 catalog_manager.cc:4841] TS 6df63b96cb5a41af9966a81a589cd065 (127.0.97.1:34801): tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (table test-table [id=31009f20f4654422977417776e181bf8]) successfully deleted
I20260929 00:54:11.572963 2074 tablet_replica.cc:330] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: stopping tablet replica
I20260929 00:54:11.573724 2075 tablet_replica.cc:330] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: stopping tablet replica
I20260929 00:54:11.574013 2074 raft_consensus.cc:2229] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:11.574374 2074 raft_consensus.cc:2258] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:11.574782 2075 raft_consensus.cc:2229] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:11.575381 2075 raft_consensus.cc:2258] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:11.576174 2074 ts_tablet_manager.cc:1905] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:11.577322 2075 ts_tablet_manager.cc:1905] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:11.585258 2075 ts_tablet_manager.cc:1918] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:11.592492 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.574255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.97.1:34801)
0929 00:54:11.574425 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.574431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.575746 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 00:54:11.575987 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.575992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.576063 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.576174 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.577715 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.578421 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.579634 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.580233 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.580336 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.585316 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.585334 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.585353 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.585390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.589178 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.590044 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.590054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.590058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.590222 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.590664 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.590671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.590677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.591099 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.591355 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.591917 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.592141 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:54:11.594043 2074 ts_tablet_manager.cc:1918] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:11.594197 2074 log.cc:1196] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/57cc2f04e67c47d5b40ab2ed99f0cc26
W20260929 00:54:11.594255 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046
I20260929 00:54:11.594731 2074 ts_tablet_manager.cc:1939] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 6deb98735585450bbad21c87d4f58645: Deleting consensus metadata
I20260929 00:54:11.594954 2075 log.cc:1196] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/57cc2f04e67c47d5b40ab2ed99f0cc26
I20260929 00:54:11.595542 2075 ts_tablet_manager.cc:1939] T 57cc2f04e67c47d5b40ab2ed99f0cc26 P 80298f953c834640a44107008b74dbe9: Deleting consensus metadata
I20260929 00:54:11.597667 1497 catalog_manager.cc:4841] TS 6deb98735585450bbad21c87d4f58645 (127.0.97.2:39507): tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (table test-table [id=31009f20f4654422977417776e181bf8]) successfully deleted
I20260929 00:54:11.598173 1495 catalog_manager.cc:4841] TS 80298f953c834640a44107008b74dbe9 (127.0.97.3:36109): tablet 57cc2f04e67c47d5b40ab2ed99f0cc26 (table test-table [id=31009f20f4654422977417776e181bf8]) successfully deleted
I20260929 00:54:11.620240 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.599920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.97.62:38367)
0929 00:54:11.601116 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.601122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.604833 (+ 3711us) server_negotiation.cc:407] Connection header received
0929 00:54:11.605022 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.605028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.605117 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.605228 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.606848 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.608098 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.609934 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.610648 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.610784 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.613656 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.613664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.613669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.613713 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.617711 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.618342 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.618348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.618353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.618505 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.618844 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.618849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.618853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.619209 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.619469 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.619653 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.619885 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:54:11.620947 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40888
I20260929 00:54:11.622841 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1582
I20260929 00:54:11.637240 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.623910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.97.1:34801)
0929 00:54:11.624101 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.624108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.624296 (+ 188us) server_negotiation.cc:407] Connection header received
0929 00:54:11.624402 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.624407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.624522 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.624632 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.625617 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.626862 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.628181 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.628920 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.629054 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.631068 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.631077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.631085 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.631130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.634664 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.635234 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.635240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.635244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.635395 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.635778 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.635784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.635788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.636122 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.636386 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.636621 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.636848 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 00:54:11.637820 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38052
W20260929 00:54:11.638339 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:11.638495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1582
I20260929 00:54:11.656958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1722
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:11.675267 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.657729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.97.2:39507)
0929 00:54:11.657964 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.657972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.658000 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:11.658074 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.658080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.658179 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.658300 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.659411 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.660866 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.662069 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.662893 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.663063 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.668092 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.668107 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.668130 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.668184 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.672442 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.673091 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.673101 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.673105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.673267 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.673590 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.673596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.673600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.674095 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.674465 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.674625 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.674872 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 00:54:11.676011 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43652
W20260929 00:54:11.676750 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:11.676920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1722
I20260929 00:54:11.696640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1861
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:11.711133 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:11.697356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.97.3:36109)
0929 00:54:11.697580 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:11.697586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:11.697614 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:11.697818 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:11.697822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:11.697897 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:11.698003 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:11.699191 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.699891 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.701230 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:11.701636 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:11.701763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:11.705851 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:11.705866 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:11.705883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:11.705912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:11.708611 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.709246 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.709253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.709256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.709379 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:11.709740 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:11.709746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:11.709749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:11.710078 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:11.710415 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:11.710601 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 00:54:11.710827 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 00:54:11.711758 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50722
W20260929 00:54:11.712349 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:11.712467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1861
I20260929 00:54:11.733238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 1479
I20260929 00:54:11.734376 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:11.847026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 1479
2026-09-29T00:54:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4620 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[2084](info): set up 2 sockets
Sep 29 00:54:11 dist-test-slave-pzsx krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260929 00:54:12.916688 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:13 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:13.093465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37819
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:13.399375 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:13.399653 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:13.399818 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:13.406854 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:13.406957 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:54:13.406997 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:13.407091 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:13.407183 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:13.407296 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:13.420751 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37819
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:13.422052 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:13.423372 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:13.435075 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:13.435232 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:13.435227 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:13.436007 2097 server_base.cc:1034] running on GCE node
I20260929 00:54:13.436837 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:13.438230 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:13.439473 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790643253439418 us; error 29 us; skew 500 ppm
I20260929 00:54:13.439797 2097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:13 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:13.442695 2097 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:13.444159 2097 webserver.cc:460] Webserver started at http://127.0.97.62:40441/ using document root <none> and password file <none>
I20260929 00:54:13.444929 2097 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:13.445057 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:13.445367 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:13.448243 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "919c9b8de821479899faa13a356e6942"
format_stamp: "Formatted at 2026-09-29 00:54:13 on dist-test-slave-pzsx"
server_key: "8f965373b44e4d4528e0a1a7b8cf316e"
server_key_iv: "626baf8074656a9c1cc9118df3a83ce6"
server_key_version: "encryptionkey@0"
I20260929 00:54:13.449046 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "919c9b8de821479899faa13a356e6942"
format_stamp: "Formatted at 2026-09-29 00:54:13 on dist-test-slave-pzsx"
server_key: "8f965373b44e4d4528e0a1a7b8cf316e"
server_key_iv: "626baf8074656a9c1cc9118df3a83ce6"
server_key_version: "encryptionkey@0"
I20260929 00:54:13.453898 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 00:54:13.457161 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:13.458410 2097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:13.458580 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "919c9b8de821479899faa13a356e6942"
format_stamp: "Formatted at 2026-09-29 00:54:13 on dist-test-slave-pzsx"
server_key: "8f965373b44e4d4528e0a1a7b8cf316e"
server_key_iv: "626baf8074656a9c1cc9118df3a83ce6"
server_key_version: "encryptionkey@0"
I20260929 00:54:13.458791 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:13.477650 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:13.481544 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:13.481812 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:13.502844 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37819
I20260929 00:54:13.502851 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37819 every 8 connection(s)
I20260929 00:54:13.504189 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:13.507781 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:13.509753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2097
I20260929 00:54:13.509954 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:13.510733 388 external_mini_cluster.cc:1428] Setting key a5bc79599e64676f02ca8b8d92e51b44
I20260929 00:54:13.520196 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Bootstrap starting.
I20260929 00:54:13.523979 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Neither blocks nor log segments found. Creating new log.
Sep 29 00:54:13 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:13.525171 2174 log.cc:824] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:13.528446 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: No bootstrap required, opened a new log
I20260929 00:54:13.534231 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:13.512773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.97.62:37819)
0929 00:54:13.513503 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:13.513515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:13.513554 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:54:13.515126 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:13.515144 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:13.515536 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:13.515870 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:13.517706 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:13.518951 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:13.520683 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:13.521273 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:13.521387 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:13.526031 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:13.526038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:13.526052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:13.526084 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:13.529873 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:13.530524 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:13.530537 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:13.530545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:13.530743 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:13.531070 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:13.531074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:13.531078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:13.531796 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:13.532113 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:13.532390 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 00:54:13.532905 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":346,"threads_started":1}
I20260929 00:54:13.537815 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } }
I20260929 00:54:13.538141 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:13.538211 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919c9b8de821479899faa13a356e6942, State: Initialized, Role: FOLLOWER
I20260929 00:54:13.538661 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } }
I20260929 00:54:13.538847 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:13.538939 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:13.539047 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:13.541185 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } }
I20260929 00:54:13.541654 2174 leader_election.cc:302] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919c9b8de821479899faa13a356e6942; no voters:
I20260929 00:54:13.542181 2174 leader_election.cc:288] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:13.542313 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:13.542994 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [term 1 LEADER]: Becoming Leader. State: Replica: 919c9b8de821479899faa13a356e6942, State: Running, Role: LEADER
I20260929 00:54:13.543489 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } }
I20260929 00:54:13.544191 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:13.546306 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919c9b8de821479899faa13a356e6942. Latest consensus state: current_term: 1 leader_uuid: "919c9b8de821479899faa13a356e6942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } } }
I20260929 00:54:13.546695 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:13.546295 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "919c9b8de821479899faa13a356e6942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919c9b8de821479899faa13a356e6942" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37819 } } }
I20260929 00:54:13.547465 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:13.549093 2185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:13.553885 2185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:13.562788 2185 catalog_manager.cc:1347] Generated new cluster ID: c20ffb2d809e4eccb2e2d8793954307f
I20260929 00:54:13.562893 2185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:13.600521 2185 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:13.602095 2185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:13.609122 2185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Generated new TSK 0
I20260929 00:54:13.609906 2185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:13.659329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37819
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:13.975538 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:13.975793 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:13.975987 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:13.983026 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:13.983158 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:13.983309 2200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:54:13.983369 2200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:13.983512 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:13.993109 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:13.994307 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:13.995623 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:14.008616 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:14.008605 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:14.009258 2200 server_base.cc:1034] running on GCE node
W20260929 00:54:14.009378 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:14.010031 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:14.010975 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:14.012204 2200 hybrid_clock.cc:648] HybridClock initialized: now 1790643254012161 us; error 62 us; skew 500 ppm
I20260929 00:54:14.012578 2200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:14 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:14.015493 2200 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:14.016950 2200 webserver.cc:460] Webserver started at http://127.0.97.1:34031/ using document root <none> and password file <none>
I20260929 00:54:14.017624 2200 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:14.017807 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:14.018121 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:14.021041 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "b7f2396acbc34405a5f863d2c0f87afa"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "57484e36feb5c9efa42e72fc2889fc1d"
server_key_iv: "ccd34e4f35bfd1e179896279a292c336"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.021773 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b7f2396acbc34405a5f863d2c0f87afa"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "57484e36feb5c9efa42e72fc2889fc1d"
server_key_iv: "ccd34e4f35bfd1e179896279a292c336"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.026579 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 00:54:14.029881 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:14.031296 2200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:14.031441 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "b7f2396acbc34405a5f863d2c0f87afa"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "57484e36feb5c9efa42e72fc2889fc1d"
server_key_iv: "ccd34e4f35bfd1e179896279a292c336"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.031625 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:14.066083 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:14.069974 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:14.070261 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:14.072741 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:14.072867 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:14.073014 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:14.073115 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:54:14 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:14.098917 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.073487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46797 (local address 127.0.97.62:37819)
0929 00:54:14.074023 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:14.074029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:14.075286 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 00:54:14.078475 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:14.078482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:14.078563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:14.078694 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:14.080909 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.082198 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.084389 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.085206 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.085334 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.090893 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:14.090903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:14.090911 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:14.090963 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:14.095216 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.096193 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.096198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.096202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.096378 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.097007 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.097010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.097012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.097270 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:14.097374 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:14.098458 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0929 00:54:14.098675 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":313,"threads_started":1}
I20260929 00:54:14.100953 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.074197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.1:46797)
0929 00:54:14.075094 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.075160 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:14.078096 (+ 2936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:14.078968 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:14.078982 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:14.079784 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:14.080700 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.080733 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.082368 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.082375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.084214 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.084234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.085516 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.085525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.085787 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.086707 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:14.086739 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:14.090534 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:14.095366 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.095373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.095386 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.096046 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.096546 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.096553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.096560 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.096886 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.097398 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:14.097409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:14.097958 (+ 549us) client_negotiation.cc:769] Sending connection context
0929 00:54:14.098340 (+ 382us) client_negotiation.cc:240] Negotiation successful
0929 00:54:14.099039 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":346,"threads_started":1}
I20260929 00:54:14.124799 2200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42465
I20260929 00:54:14.124799 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42465 every 8 connection(s)
I20260929 00:54:14.126554 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:14.127957 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.111163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48249 (local address 127.0.97.62:37819)
0929 00:54:14.111363 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:14.111372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:14.112814 (+ 1442us) server_negotiation.cc:407] Connection header received
0929 00:54:14.112875 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:14.112881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:14.112966 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:14.113063 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:14.114600 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.115848 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.117504 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.118251 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.118396 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.121210 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:14.121219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:14.121227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:14.121274 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:14.124876 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.125697 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.125703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.125706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.125848 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.126245 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.126250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.126253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.126644 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:14.126776 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:14.127383 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 00:54:14.127614 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 00:54:14.128425 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.111168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.1:48249)
0929 00:54:14.112482 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.112522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:14.112683 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:14.113265 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:14.113270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:14.113944 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:14.114396 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.114420 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.116089 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.116095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.117313 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.117325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.118453 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.118462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.118695 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.119499 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:14.119526 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:14.120967 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:14.125050 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.125056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.125060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.125550 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.125986 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.125989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.125991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.126100 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.126833 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:14.126839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:14.127046 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 00:54:14.127250 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 00:54:14.127771 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":725,"threads_started":1}
I20260929 00:54:14.129038 2336 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37819
I20260929 00:54:14.129364 2336 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:14.130055 2336 heartbeater.cc:511] Master 127.0.97.62:37819 requested a full tablet report, sending...
I20260929 00:54:14.132112 2129 ts_manager.cc:194] Registered new tserver with Master: b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465)
I20260929 00:54:14.133884 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46797
I20260929 00:54:14.136168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2200
I20260929 00:54:14.136375 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:14.137107 388 external_mini_cluster.cc:1428] Setting key 7d62641cd49fe3c58e0458d602a3d637
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:14.173961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37819
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:14.488699 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:14.489001 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:14.489202 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:14.496176 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:14.496270 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:14.496380 2340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:54:14.496459 2340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:14.496615 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:14.508641 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:14.510133 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:14.511669 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:14.525483 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:14.525629 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:14.525799 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:14.526572 2340 server_base.cc:1034] running on GCE node
I20260929 00:54:14.527145 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:14.527988 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:14.529179 2340 hybrid_clock.cc:648] HybridClock initialized: now 1790643254529132 us; error 42 us; skew 500 ppm
I20260929 00:54:14.529506 2340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:14 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:14.532488 2340 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:14.534003 2340 webserver.cc:460] Webserver started at http://127.0.97.2:39469/ using document root <none> and password file <none>
I20260929 00:54:14.534663 2340 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:14.534788 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:14.535102 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:14.537989 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "686ef0199904e8f4a3d7995be6cec583"
server_key_iv: "ca246a9b2e1ec332b4f35fc470329cae"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.538738 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "686ef0199904e8f4a3d7995be6cec583"
server_key_iv: "ca246a9b2e1ec332b4f35fc470329cae"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.543502 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 00:54:14.546965 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:14.548216 2340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:54:14.548372 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646"
format_stamp: "Formatted at 2026-09-29 00:54:14 on dist-test-slave-pzsx"
server_key: "686ef0199904e8f4a3d7995be6cec583"
server_key_iv: "ca246a9b2e1ec332b4f35fc470329cae"
server_key_version: "encryptionkey@0"
I20260929 00:54:14.548617 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:14.588433 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:14.592919 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:14.593164 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:14.595598 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:14.595741 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:14.595854 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:14.595935 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:54:14 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:14.619683 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.596402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52437 (local address 127.0.97.62:37819)
0929 00:54:14.596585 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:14.596605 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:14.597982 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 00:54:14.599961 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:14.599967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:14.600048 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:14.600170 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:14.602402 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.603570 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.606233 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.606967 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.607146 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.612342 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:14.612347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:14.612350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:14.612381 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:14.616116 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.617128 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.617134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.617139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.617299 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.617980 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.617985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.617990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.618296 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:14.618443 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:14.619138 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 00:54:14.619337 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:54:14.620826 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.597040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.2:52437)
0929 00:54:14.597823 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.597861 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:14.599649 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:14.600444 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:14.600457 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:14.601295 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:14.602208 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.602242 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.603718 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.603723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.606054 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.606074 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.607238 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.607248 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.607543 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.608327 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:14.608346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:14.612093 (+ 3747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:14.616286 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.616303 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.616320 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.617017 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.617432 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.617440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.617447 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.617841 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.618424 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:14.618432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:14.618754 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 00:54:14.619032 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 00:54:14.619523 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":362,"threads_started":1}
I20260929 00:54:14.642735 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.626263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.2:60973)
0929 00:54:14.626997 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:14.627021 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:14.627176 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:14.627473 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:14.627479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:14.628176 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:14.628755 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.628773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.630691 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.630699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.632807 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:14.632825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.633689 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.633698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:14.633956 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.634619 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:14.634649 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:14.635881 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:14.639875 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.639884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.639888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.640553 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.641085 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:14.641088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:14.641091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:14.641202 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:14.641954 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:14.641957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:14.642094 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 00:54:14.642205 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:54:14.642414 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":383,"threads_started":1}
I20260929 00:54:14.643101 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:14.626311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60973 (local address 127.0.97.62:37819)
0929 00:54:14.626526 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:14.626534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:14.627117 (+ 583us) server_negotiation.cc:407] Connection header received
0929 00:54:14.627294 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:14.627297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:14.627351 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:14.627418 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:14.629050 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.630463 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.632984 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:14.633480 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:14.633621 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:14.636058 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:14.636062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:14.636065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:14.636090 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:14.639733 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.640760 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.640767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.640774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.640957 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:14.641314 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:14.641321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:14.641326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:14.641841 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:14.642012 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:14.642363 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 00:54:14.642643 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:54:14.653702 2340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45369
I20260929 00:54:14.653753 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45369 every 8 connection(s)
I20260929 00:54:14.654850 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:14.656677 2475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37819
I20260929 00:54:14.656875 2475 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:14.657460 2475 heartbeater.cc:511] Master 127.0.97.62:37819 requested a full tablet report, sending...
I20260929 00:54:14.658676 2129 ts_manager.cc:194] Registered new tserver with Master: 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369)
I20260929 00:54:14.659500 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52437
I20260929 00:54:14.661139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2340
I20260929 00:54:14.661295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:14.661744 388 external_mini_cluster.cc:1428] Setting key 4244da33b32ec2de89fdb371cce4efa9
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:14.697623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37819
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:15.010458 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:15.010725 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:15.010824 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:15.017964 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:15.018072 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:15.018157 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 00:54:15.018208 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:54:15.018268 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:15.027949 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:15.029160 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:15.030375 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:15.043089 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:15.043061 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:15.043464 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:15.043799 2479 server_base.cc:1034] running on GCE node
I20260929 00:54:15.044566 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:15.045589 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:15.046815 2479 hybrid_clock.cc:648] HybridClock initialized: now 1790643255046762 us; error 47 us; skew 500 ppm
I20260929 00:54:15.047125 2479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:15 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:15.053777 2479 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:15.055209 2479 webserver.cc:460] Webserver started at http://127.0.97.3:39339/ using document root <none> and password file <none>
I20260929 00:54:15.055830 2479 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:15.055927 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:15.056216 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:15.059088 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "dd8e9ddc18b3411b973c11ee35f68440"
format_stamp: "Formatted at 2026-09-29 00:54:15 on dist-test-slave-pzsx"
server_key: "a2a3133ad080eb084f676cc3dfb92d5a"
server_key_iv: "bcf002a114104de58374ec48b12d844a"
server_key_version: "encryptionkey@0"
I20260929 00:54:15.059782 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dd8e9ddc18b3411b973c11ee35f68440"
format_stamp: "Formatted at 2026-09-29 00:54:15 on dist-test-slave-pzsx"
server_key: "a2a3133ad080eb084f676cc3dfb92d5a"
server_key_iv: "bcf002a114104de58374ec48b12d844a"
server_key_version: "encryptionkey@0"
I20260929 00:54:15.064778 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 00:54:15.067919 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:15.069211 2479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:15.069329 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "dd8e9ddc18b3411b973c11ee35f68440"
format_stamp: "Formatted at 2026-09-29 00:54:15 on dist-test-slave-pzsx"
server_key: "a2a3133ad080eb084f676cc3dfb92d5a"
server_key_iv: "bcf002a114104de58374ec48b12d844a"
server_key_version: "encryptionkey@0"
I20260929 00:54:15.069480 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:15.083751 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:15.087402 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:15.087635 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:15.090473 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:15.090566 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:15.090729 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:15.090842 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:54:15 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:15.114809 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.091078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55231 (local address 127.0.97.62:37819)
0929 00:54:15.091285 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.091291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.092752 (+ 1461us) server_negotiation.cc:407] Connection header received
0929 00:54:15.095725 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.095733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.095826 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.095930 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:15.098118 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.099431 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.101758 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.102553 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.102727 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.107383 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:15.107390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:15.107397 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:15.107436 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:15.110858 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.111798 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.111803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.111807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.111937 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.112359 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.112363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.112367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.112782 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:15.112950 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.113977 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.114244 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:54:15.115585 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.091647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.3:55231)
0929 00:54:15.092559 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.092621 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.095353 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.096243 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.096260 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.097092 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:15.097900 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.097944 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.099605 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.099613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.101552 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.101615 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.102829 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.102836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.103056 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.103824 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:15.103851 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:15.107102 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:15.111000 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.111009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.111025 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.111673 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.112042 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.112047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.112050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.112247 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.113020 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:15.113029 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:15.113499 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.113834 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.114400 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":394,"threads_started":1}
I20260929 00:54:15.136942 2336 heartbeater.cc:503] Master 127.0.97.62:37819 was elected leader, sending a full tablet report...
I20260929 00:54:15.142915 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.121354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.97.3:55951)
0929 00:54:15.127929 (+ 6575us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.127951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.128095 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.128989 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.128994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.129670 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:15.130074 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.130085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.131473 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.131479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.133265 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.133279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.134162 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.134170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.134437 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.135182 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:15.135199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:15.136365 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:15.139869 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.139876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.139880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.140251 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.140682 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.140687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.140691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.140831 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.141529 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:15.141537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:15.141724 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.141899 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.142282 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6433,"thread_start_us":6821,"threads_started":1}
I20260929 00:54:15.143535 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.121332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55951 (local address 127.0.97.62:37819)
0929 00:54:15.121536 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.121543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.128057 (+ 6514us) server_negotiation.cc:407] Connection header received
0929 00:54:15.128609 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.128615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.128705 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.128774 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:15.130248 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.131304 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.133449 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.133887 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.134044 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.136637 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:15.136645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:15.136652 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:15.136715 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:15.139741 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.140375 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.140380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.140383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.140568 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.140969 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.140974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.140977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.141288 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:15.141461 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.142671 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.142884 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:54:15.152323 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36537
I20260929 00:54:15.152392 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36537 every 8 connection(s)
I20260929 00:54:15.153839 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:15.155956 2614 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37819
I20260929 00:54:15.156189 2614 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:15.156886 2614 heartbeater.cc:511] Master 127.0.97.62:37819 requested a full tablet report, sending...
I20260929 00:54:15.158181 2129 ts_manager.cc:194] Registered new tserver with Master: dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:15.158958 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55231
I20260929 00:54:15.163761 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2479
I20260929 00:54:15.163923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:15.164353 388 external_mini_cluster.cc:1428] Setting key 88893910faaac122654d46e9f5930770
I20260929 00:54:15.167073 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:54:15 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:54:15 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:15.211115 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.198676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.0.97.62:37819)
0929 00:54:15.198898 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.198904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.199076 (+ 172us) server_negotiation.cc:407] Connection header received
0929 00:54:15.199224 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.199228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.199307 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.199394 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:15.200739 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.201445 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.202548 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.202965 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.203054 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.206653 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:15.206658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:15.206661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:15.206693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:15.209031 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.209547 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.209550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.209553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.209676 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.209970 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.209973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.209975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.210198 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:15.210358 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.210592 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.210738 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:54:15.214850 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:15.221704 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:15.255903 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.234090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.97.2:45369)
0929 00:54:15.243866 (+ 9776us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.243879 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.247052 (+ 3173us) server_negotiation.cc:407] Connection header received
0929 00:54:15.247199 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.247210 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.247544 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.247775 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.249009 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.250175 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.252918 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.254257 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.254393 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.254589 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.254903 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.255100 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9551,"thread_start_us":9379,"threads_started":1}
I20260929 00:54:15.256065 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.233695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42465 (local address 127.0.0.1:41784)
0929 00:54:15.247664 (+ 13969us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.247681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.247785 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.249205 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.249211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.249239 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.249604 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.249617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.251471 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.251478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.253682 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.253691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.255265 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.255271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.255455 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.255500 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.255568 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.255719 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13869,"mutex_wait_us":37,"thread_start_us":14286,"threads_started":1}
I20260929 00:54:15.256228 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.233242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36537 (local address 127.0.0.1:48520)
0929 00:54:15.234146 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.234190 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.234338 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.248056 (+ 13718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.248067 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.248101 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.249130 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.249145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.251132 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.251137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.253365 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.253379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.255075 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.255081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.255249 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.255279 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.255350 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.255489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":1,"thread_start_us":736,"threads_started":1}
I20260929 00:54:15.256385 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.233770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.0.97.3:36537)
0929 00:54:15.247425 (+ 13655us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.247436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.247472 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:15.247582 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.247590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.247829 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.248267 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.249335 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.250990 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.253529 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.254863 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.255633 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.255856 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.255926 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.256095 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13491,"thread_start_us":13334,"threads_started":1}
I20260929 00:54:15.255519 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.233310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45369 (local address 127.0.0.1:60562)
0929 00:54:15.246686 (+ 13376us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.246712 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.246890 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.248115 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.248119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.248140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.248704 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.248719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.250306 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.250314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.252750 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.252763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.254436 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.254443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.254631 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.254687 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.254814 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.254984 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13188,"mutex_wait_us":3,"thread_start_us":14566,"threads_started":1}
I20260929 00:54:15.257205 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.234491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.97.1:42465)
0929 00:54:15.248580 (+ 14089us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.248586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.248633 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:54:15.248723 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.248732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.248975 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.249804 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.249985 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.251331 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.253852 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.255043 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.256637 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.256846 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.256911 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.257018 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13917,"thread_start_us":13777,"threads_started":1}
I20260929 00:54:15.259872 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 0377017670e9492db1fcf1786c287128 (DEFAULT_TABLE table=test-table [id=3cf6e13a19bc44daa319b81c33d4a559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.259865 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 0377017670e9492db1fcf1786c287128 (DEFAULT_TABLE table=test-table [id=3cf6e13a19bc44daa319b81c33d4a559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.261271 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0377017670e9492db1fcf1786c287128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.261291 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0377017670e9492db1fcf1786c287128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.262530 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 0377017670e9492db1fcf1786c287128 (DEFAULT_TABLE table=test-table [id=3cf6e13a19bc44daa319b81c33d4a559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.263849 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0377017670e9492db1fcf1786c287128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.272464 2633 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Bootstrap starting.
I20260929 00:54:15.275638 2633 tablet_bootstrap.cc:654] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.276407 2634 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Bootstrap starting.
I20260929 00:54:15.276930 2633 log.cc:824] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:15.277909 2635 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Bootstrap starting.
I20260929 00:54:15.280702 2634 tablet_bootstrap.cc:654] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.280753 2635 tablet_bootstrap.cc:654] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.281239 2633 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: No bootstrap required, opened a new log
I20260929 00:54:15.281498 2633 ts_tablet_manager.cc:1397] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 00:54:15.281796 2635 log.cc:824] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:15.281863 2634 log.cc:824] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:15.284353 2634 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: No bootstrap required, opened a new log
I20260929 00:54:15.284612 2635 tablet_bootstrap.cc:492] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: No bootstrap required, opened a new log
I20260929 00:54:15.284643 2634 ts_tablet_manager.cc:1397] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 00:54:15.284849 2635 ts_tablet_manager.cc:1397] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260929 00:54:15.291745 2634 raft_consensus.cc:348] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.291791 2633 raft_consensus.cc:348] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.292088 2634 raft_consensus.cc:374] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.292201 2634 raft_consensus.cc:729] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.292258 2633 raft_consensus.cc:374] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.292366 2633 raft_consensus.cc:729] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.292847 2634 consensus_queue.cc:260] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.293066 2633 consensus_queue.cc:260] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.293884 2475 heartbeater.cc:503] Master 127.0.97.62:37819 was elected leader, sending a full tablet report...
I20260929 00:54:15.294517 2634 ts_tablet_manager.cc:1428] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260929 00:54:15.295395 2635 raft_consensus.cc:348] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.295931 2635 raft_consensus.cc:374] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.296051 2635 raft_consensus.cc:729] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8e9ddc18b3411b973c11ee35f68440, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.296748 2635 consensus_queue.cc:260] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.301810 2633 ts_tablet_manager.cc:1428] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Time spent starting tablet: real 0.020s user 0.018s sys 0.001s
I20260929 00:54:15.302737 2614 heartbeater.cc:503] Master 127.0.97.62:37819 was elected leader, sending a full tablet report...
I20260929 00:54:15.303390 2635 ts_tablet_manager.cc:1428] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
W20260929 00:54:15.384352 2337 tablet.cc:2369] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:15.405400 2615 tablet.cc:2369] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:15.406729 2476 tablet.cc:2369] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:15.420063 2640 raft_consensus.cc:482] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:15.420341 2640 raft_consensus.cc:504] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.422062 2640 leader_election.cc:288] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537), 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369)
I20260929 00:54:15.428699 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.422391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52667 (local address 127.0.97.3:36537)
0929 00:54:15.422645 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.422651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.423454 (+ 803us) server_negotiation.cc:407] Connection header received
0929 00:54:15.423679 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.423683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.423739 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.423832 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.424437 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.425227 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.427002 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.427740 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.427893 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.428002 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.428288 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.428407 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 00:54:15.428727 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.422479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33715 (local address 127.0.97.2:45369)
0929 00:54:15.422670 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.422678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.423547 (+ 869us) server_negotiation.cc:407] Connection header received
0929 00:54:15.423738 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.423743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.423826 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.423930 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.424589 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.425326 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.426999 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.427750 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.427907 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.428018 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.428325 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.428445 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:54:15.428853 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.422461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36537 (local address 127.0.97.1:52667)
0929 00:54:15.423349 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.423364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.423536 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.423845 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.423849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.423870 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.424294 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.424307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.425346 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.425353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.426847 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.426858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.427963 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.427969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.428136 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.428158 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.428263 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.428386 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":3,"thread_start_us":529,"threads_started":1}
I20260929 00:54:15.428906 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.422485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45369 (local address 127.0.97.1:33715)
0929 00:54:15.423441 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.423454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.423599 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.423944 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.423948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.423983 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.424413 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.424422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.425459 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.425465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.426819 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.426848 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.428006 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.428012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.428178 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.428199 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.428301 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.428418 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":23,"thread_start_us":611,"threads_started":1}
I20260929 00:54:15.430182 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0377017670e9492db1fcf1786c287128" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" is_pre_election: true
I20260929 00:54:15.430181 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0377017670e9492db1fcf1786c287128" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440" is_pre_election: true
I20260929 00:54:15.430719 2430 raft_consensus.cc:2454] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 0.
I20260929 00:54:15.430720 2568 raft_consensus.cc:2454] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 0.
I20260929 00:54:15.431412 2217 leader_election.cc:302] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.431824 2640 raft_consensus.cc:2781] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:15.431967 2640 raft_consensus.cc:482] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:15.432076 2640 raft_consensus.cc:3037] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.434371 2640 raft_consensus.cc:504] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.434933 2640 leader_election.cc:288] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 election: Requested vote from peers dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537), 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369)
I20260929 00:54:15.435364 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0377017670e9492db1fcf1786c287128" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440"
I20260929 00:54:15.435563 2568 raft_consensus.cc:3037] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.435539 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0377017670e9492db1fcf1786c287128" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646"
I20260929 00:54:15.435751 2430 raft_consensus.cc:3037] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.438189 2568 raft_consensus.cc:2454] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 1.
I20260929 00:54:15.438550 2217 leader_election.cc:302] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.438784 2430 raft_consensus.cc:2454] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 1.
I20260929 00:54:15.438809 2640 raft_consensus.cc:2781] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:15.439291 2640 raft_consensus.cc:686] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 1 LEADER]: Becoming Leader. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Running, Role: LEADER
I20260929 00:54:15.439949 2640 consensus_queue.cc:237] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } }
I20260929 00:54:15.444113 2129 catalog_manager.cc:5495] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } health_report { overall_health: HEALTHY } } }
Sep 29 00:54:15 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:15.527719 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.514933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37819 (local address 127.0.0.1:48074)
0929 00:54:15.515517 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.515533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.515720 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.516052 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.516056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.516786 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:15.517041 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.517050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.518090 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.518096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.519100 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.519112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.519815 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.519821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.519983 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.520668 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:15.520689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:15.522873 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:15.525721 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.525730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.525736 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.526124 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.526427 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:15.526430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:15.526432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:15.526562 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:15.526970 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:15.526976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:15.527126 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.527250 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.527427 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":332,"threads_started":1}
I20260929 00:54:15.527748 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.514867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.0.97.62:37819)
0929 00:54:15.515043 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.515047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.515629 (+ 582us) server_negotiation.cc:407] Connection header received
0929 00:54:15.515846 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.515850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.515910 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.516022 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:15.517175 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.517980 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.519226 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.519646 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.519777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.523030 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:15.523034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:15.523038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:15.523064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:15.525589 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.526218 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.526222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.526224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.526340 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:15.526648 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:15.526651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:15.526653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:15.526872 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:15.526970 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.527335 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.527515 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":67}
I20260929 00:54:15.532140 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48074:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:54:15.533083 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:15.539314 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 5f68b284162b44e39876ee2a670e8f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd4dc57039c74fdeb977eac52dcd9cbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:15.539212 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 5f68b284162b44e39876ee2a670e8f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd4dc57039c74fdeb977eac52dcd9cbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:15.539631 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 5f68b284162b44e39876ee2a670e8f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd4dc57039c74fdeb977eac52dcd9cbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:54:15.540196 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f68b284162b44e39876ee2a670e8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.540225 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f68b284162b44e39876ee2a670e8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.540371 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f68b284162b44e39876ee2a670e8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.545893 2634 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Bootstrap starting.
I20260929 00:54:15.547224 2635 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: Bootstrap starting.
I20260929 00:54:15.547272 2634 tablet_bootstrap.cc:654] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.548753 2633 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: Bootstrap starting.
I20260929 00:54:15.548907 2635 tablet_bootstrap.cc:654] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.549438 2634 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: No bootstrap required, opened a new log
I20260929 00:54:15.549592 2634 ts_tablet_manager.cc:1397] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 00:54:15.550191 2634 raft_consensus.cc:348] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.550395 2634 raft_consensus.cc:374] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.550484 2634 raft_consensus.cc:729] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.550554 2633 tablet_bootstrap.cc:654] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.550730 2634 consensus_queue.cc:260] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.551360 2634 ts_tablet_manager.cc:1428] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 00:54:15.551234 2639 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.551658 2639 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.559449 2633 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: No bootstrap required, opened a new log
I20260929 00:54:15.559619 2633 ts_tablet_manager.cc:1397] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260929 00:54:15.559952 2635 tablet_bootstrap.cc:492] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: No bootstrap required, opened a new log
I20260929 00:54:15.560091 2635 ts_tablet_manager.cc:1397] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 00:54:15.560418 2633 raft_consensus.cc:348] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.560686 2633 raft_consensus.cc:374] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.560788 2633 raft_consensus.cc:729] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.560900 2635 raft_consensus.cc:348] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.561146 2635 raft_consensus.cc:374] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.561017 2633 consensus_queue.cc:260] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.561233 2635 raft_consensus.cc:729] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8e9ddc18b3411b973c11ee35f68440, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.561461 2640 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.561650 2633 ts_tablet_manager.cc:1428] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:15.561517 2635 consensus_queue.cc:260] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.561760 2640 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.562117 2641 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.562467 2641 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.562609 2635 ts_tablet_manager.cc:1428] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:15.574600 2640 raft_consensus.cc:482] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:15.574752 2640 raft_consensus.cc:504] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.575331 2640 leader_election.cc:288] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:15.575762 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f68b284162b44e39876ee2a670e8f0c" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" is_pre_election: true
I20260929 00:54:15.575798 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f68b284162b44e39876ee2a670e8f0c" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440" is_pre_election: true
I20260929 00:54:15.576020 2430 raft_consensus.cc:2454] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 0.
I20260929 00:54:15.576020 2568 raft_consensus.cc:2454] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 0.
I20260929 00:54:15.576428 2217 leader_election.cc:302] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.576771 2640 raft_consensus.cc:2781] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:15.576876 2640 raft_consensus.cc:482] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:15.576984 2640 raft_consensus.cc:3037] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.579231 2640 raft_consensus.cc:504] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.579833 2640 leader_election.cc:288] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 election: Requested vote from peers 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:15.580227 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f68b284162b44e39876ee2a670e8f0c" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440"
I20260929 00:54:15.580312 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f68b284162b44e39876ee2a670e8f0c" candidate_uuid: "b7f2396acbc34405a5f863d2c0f87afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646"
I20260929 00:54:15.580406 2568 raft_consensus.cc:3037] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.580582 2430 raft_consensus.cc:3037] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.582273 2568 raft_consensus.cc:2454] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 1.
I20260929 00:54:15.582587 2430 raft_consensus.cc:2454] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f2396acbc34405a5f863d2c0f87afa in term 1.
I20260929 00:54:15.582662 2217 leader_election.cc:302] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.582922 2640 raft_consensus.cc:2781] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:15.583137 2640 raft_consensus.cc:686] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [term 1 LEADER]: Becoming Leader. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Running, Role: LEADER
I20260929 00:54:15.583362 2640 consensus_queue.cc:237] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.584924 2644 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: TxnStatusTablet state changed. Reason: New leader b7f2396acbc34405a5f863d2c0f87afa. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.585229 2644 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:15.585919 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:54:15.586205 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:54:15.586225 2127 catalog_manager.cc:5495] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:15.603415 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.598056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42465 (local address 127.0.0.1:41794)
0929 00:54:15.598292 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.598307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.598457 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.598895 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.598899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.598910 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:15.599084 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.599093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.600060 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.600068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.601107 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.601118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.601846 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.601852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.602018 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.602066 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.602880 (+ 814us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.602901 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.602982 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.603125 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 00:54:15.603399 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.598028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.97.1:42465)
0929 00:54:15.598202 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.598209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.598372 (+ 163us) server_negotiation.cc:407] Connection header received
0929 00:54:15.598650 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.598654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.598745 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.598884 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.599211 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.599927 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.601236 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.601637 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.601823 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.602261 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.602766 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.602915 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.602998 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.603073 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 00:54:15.611093 2568 raft_consensus.cc:1261] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Refusing update from remote peer b7f2396acbc34405a5f863d2c0f87afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.611243 2430 raft_consensus.cc:1261] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Refusing update from remote peer b7f2396acbc34405a5f863d2c0f87afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.612057 2644 consensus_queue.cc:1035] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.612411 2640 consensus_queue.cc:1035] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.620743 2639 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: TxnStatusTablet state changed. Reason: New leader b7f2396acbc34405a5f863d2c0f87afa. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.621012 2639 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.625659 2639 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.625873 2639 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P 0ea3dbd3537b4d0d8f9295f7c4ab8646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.628916 2640 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.629220 2640 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:15.631106 2641 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: TxnStatusTablet state changed. Reason: New leader b7f2396acbc34405a5f863d2c0f87afa. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.631348 2641 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.632455 2644 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.633112 2644 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P b7f2396acbc34405a5f863d2c0f87afa: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:54:15.644865 2641 tablet_replica.cc:439] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7f2396acbc34405a5f863d2c0f87afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } } }
I20260929 00:54:15.645200 2641 tablet_replica.cc:442] T 5f68b284162b44e39876ee2a670e8f0c P dd8e9ddc18b3411b973c11ee35f68440: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:54:15.648180 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.640050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.97.1:42465)
0929 00:54:15.640262 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.640271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.640293 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:54:15.640352 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.640356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.640433 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.640556 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.641272 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.642812 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.646158 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.646865 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.646959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.647490 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.647609 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.647676 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.647834 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.647894 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:54:15.674117 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.666769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55689 (local address 127.0.97.1:42465)
0929 00:54:15.666940 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.666947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.667521 (+ 574us) server_negotiation.cc:407] Connection header received
0929 00:54:15.667712 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.667717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.667791 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.667877 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.668533 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.669706 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.671812 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.672479 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.672620 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.673218 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.673378 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.673474 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.673627 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.673759 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:54:15.674227 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.666769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42465 (local address 127.0.97.1:55689)
0929 00:54:15.667412 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.667432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.667580 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.667908 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.667913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.667926 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:15.668341 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.668352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.669844 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.669850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.671662 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.671677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.672795 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.672801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.672958 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.673021 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.673486 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.673495 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.673578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.673727 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":3,"thread_start_us":347,"threads_started":1}
I20260929 00:54:15.684074 2568 raft_consensus.cc:1261] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Refusing update from remote peer b7f2396acbc34405a5f863d2c0f87afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.684074 2430 raft_consensus.cc:1261] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Refusing update from remote peer b7f2396acbc34405a5f863d2c0f87afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.684736 2640 consensus_queue.cc:1035] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.685017 2666 consensus_queue.cc:1035] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.690148 2659 mvcc.cc:205] Tried to move back new op lower bound from 7334474775277207552 to 7334474774285864960. Current Snapshot: MvccSnapshot[applied={T|T < 7334474775277207552}]
I20260929 00:54:15.742801 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:15.743027 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:15.746443 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Sending DeleteTablet for 3 replicas of tablet 0377017670e9492db1fcf1786c287128
I20260929 00:54:15.747632 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0377017670e9492db1fcf1786c287128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260929 00:54:15.747686 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0377017670e9492db1fcf1786c287128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260929 00:54:15.747740 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 0377017670e9492db1fcf1786c287128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41784
I20260929 00:54:15.749891 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:15.750511 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:15.757926 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 811ade914daa427a978a410c5d195f6d (DEFAULT_TABLE table=test-table [id=511906597a4a475fa659bac971708402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.757936 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 811ade914daa427a978a410c5d195f6d (DEFAULT_TABLE table=test-table [id=511906597a4a475fa659bac971708402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.758204 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 811ade914daa427a978a410c5d195f6d (DEFAULT_TABLE table=test-table [id=511906597a4a475fa659bac971708402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:15.758447 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811ade914daa427a978a410c5d195f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.758472 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811ade914daa427a978a410c5d195f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.758599 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811ade914daa427a978a410c5d195f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:15.764731 2635 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Bootstrap starting.
I20260929 00:54:15.766876 2635 tablet_bootstrap.cc:654] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.767695 2633 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Bootstrap starting.
I20260929 00:54:15.769045 2634 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Bootstrap starting.
I20260929 00:54:15.769645 2633 tablet_bootstrap.cc:654] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.771014 2634 tablet_bootstrap.cc:654] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:15.778546 2635 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: No bootstrap required, opened a new log
I20260929 00:54:15.778800 2635 ts_tablet_manager.cc:1397] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260929 00:54:15.779785 2635 raft_consensus.cc:348] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.780220 2635 raft_consensus.cc:374] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.780351 2635 raft_consensus.cc:729] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8e9ddc18b3411b973c11ee35f68440, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.780786 2635 consensus_queue.cc:260] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.782358 2635 ts_tablet_manager.cc:1428] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:15.783061 2676 tablet_replica.cc:330] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: stopping tablet replica
I20260929 00:54:15.785979 2633 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: No bootstrap required, opened a new log
I20260929 00:54:15.785930 2676 raft_consensus.cc:2229] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:15.786157 2633 ts_tablet_manager.cc:1397] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260929 00:54:15.786506 2676 raft_consensus.cc:2258] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:15.787325 2677 tablet_replica.cc:330] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: stopping tablet replica
I20260929 00:54:15.787266 2633 raft_consensus.cc:348] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.787521 2633 raft_consensus.cc:374] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.787612 2633 raft_consensus.cc:729] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.787894 2633 consensus_queue.cc:260] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.789324 2676 ts_tablet_manager.cc:1905] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:15.790714 2633 ts_tablet_manager.cc:1428] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260929 00:54:15.792934 2677 raft_consensus.cc:2229] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:15.793113 2678 tablet_replica.cc:330] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: stopping tablet replica
I20260929 00:54:15.793434 2677 raft_consensus.cc:2258] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:15.793483 2678 raft_consensus.cc:2229] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:15.793826 2634 tablet_bootstrap.cc:492] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: No bootstrap required, opened a new log
I20260929 00:54:15.793993 2634 ts_tablet_manager.cc:1397] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260929 00:54:15.794071 2678 raft_consensus.cc:2258] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:15.794802 2634 raft_consensus.cc:348] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.795045 2634 raft_consensus.cc:374] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:15.795145 2634 raft_consensus.cc:729] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Initialized, Role: FOLLOWER
I20260929 00:54:15.795403 2634 consensus_queue.cc:260] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.795766 2678 ts_tablet_manager.cc:1905] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:15.795768 2677 ts_tablet_manager.cc:1905] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:15.795975 2634 ts_tablet_manager.cc:1428] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:15.797796 2676 ts_tablet_manager.cc:1918] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:15.797946 2676 log.cc:1196] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/0377017670e9492db1fcf1786c287128
I20260929 00:54:15.798476 2676 ts_tablet_manager.cc:1939] T 0377017670e9492db1fcf1786c287128 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting consensus metadata
I20260929 00:54:15.800611 2115 catalog_manager.cc:4841] TS 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369): tablet 0377017670e9492db1fcf1786c287128 (table test-table [id=3cf6e13a19bc44daa319b81c33d4a559]) successfully deleted
I20260929 00:54:15.803102 2678 ts_tablet_manager.cc:1918] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:15.803128 2677 ts_tablet_manager.cc:1918] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:15.803263 2677 log.cc:1196] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/0377017670e9492db1fcf1786c287128
I20260929 00:54:15.803282 2678 log.cc:1196] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/0377017670e9492db1fcf1786c287128
I20260929 00:54:15.803805 2677 ts_tablet_manager.cc:1939] T 0377017670e9492db1fcf1786c287128 P dd8e9ddc18b3411b973c11ee35f68440: Deleting consensus metadata
I20260929 00:54:15.803893 2678 ts_tablet_manager.cc:1939] T 0377017670e9492db1fcf1786c287128 P b7f2396acbc34405a5f863d2c0f87afa: Deleting consensus metadata
I20260929 00:54:15.805522 2114 catalog_manager.cc:4841] TS dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537): tablet 0377017670e9492db1fcf1786c287128 (table test-table [id=3cf6e13a19bc44daa319b81c33d4a559]) successfully deleted
I20260929 00:54:15.805588 2113 catalog_manager.cc:4841] TS b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465): tablet 0377017670e9492db1fcf1786c287128 (table test-table [id=3cf6e13a19bc44daa319b81c33d4a559]) successfully deleted
I20260929 00:54:15.809588 2639 raft_consensus.cc:482] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:15.809782 2639 raft_consensus.cc:504] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.811448 2639 leader_election.cc:288] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:15.819514 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.812004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40949 (local address 127.0.97.1:42465)
0929 00:54:15.812181 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.812188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.812318 (+ 130us) server_negotiation.cc:407] Connection header received
0929 00:54:15.812654 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.812660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.812730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.812790 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.813591 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.814388 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.816938 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.818226 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.818354 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.818475 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.819070 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.819209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:54:15.819849 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.811803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36537 (local address 127.0.97.2:46321)
0929 00:54:15.812771 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.812785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.812916 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.813217 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.813221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.813245 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.813668 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.813683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.815117 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.815123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.817176 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.817185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.819004 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.819012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.819198 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.819230 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.819339 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.819462 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":23,"thread_start_us":737,"threads_started":1}
I20260929 00:54:15.820212 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.812160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46321 (local address 127.0.97.3:36537)
0929 00:54:15.812301 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.812308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.812867 (+ 559us) server_negotiation.cc:407] Connection header received
0929 00:54:15.813041 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.813052 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.813107 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.813202 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:15.813840 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.814954 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.817339 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.818273 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.818385 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.818475 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.819375 (+ 900us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.819493 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:54:15.820245 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.811692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42465 (local address 127.0.97.2:40949)
0929 00:54:15.812220 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.812242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.812428 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.812996 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.813001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.813029 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:15.813407 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.813419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.814513 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.814521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.816772 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.816790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.818657 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.818664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.818840 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.818874 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.819736 (+ 862us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.819875 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":3,"thread_start_us":493,"threads_started":1}
I20260929 00:54:15.820818 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811ade914daa427a978a410c5d195f6d" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440" is_pre_election: true
I20260929 00:54:15.821134 2563 raft_consensus.cc:2454] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:15.821064 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811ade914daa427a978a410c5d195f6d" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa" is_pre_election: true
I20260929 00:54:15.821506 2289 raft_consensus.cc:2454] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:15.821658 2357 leader_election.cc:302] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.822043 2639 raft_consensus.cc:2781] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:15.822165 2639 raft_consensus.cc:482] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:15.822278 2639 raft_consensus.cc:3037] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.825326 2639 raft_consensus.cc:504] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.825968 2639 leader_election.cc:288] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Requested vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:15.826387 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811ade914daa427a978a410c5d195f6d" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440"
I20260929 00:54:15.826431 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811ade914daa427a978a410c5d195f6d" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa"
I20260929 00:54:15.826576 2563 raft_consensus.cc:3037] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.826648 2289 raft_consensus.cc:3037] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:15.828558 2563 raft_consensus.cc:2454] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:15.828953 2357 leader_election.cc:302] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:15.829223 2639 raft_consensus.cc:2781] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:15.829721 2639 raft_consensus.cc:686] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Running, Role: LEADER
I20260929 00:54:15.829901 2289 raft_consensus.cc:2454] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:15.830111 2639 consensus_queue.cc:237] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:15.833662 2129 catalog_manager.cc:5495] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:15.866030 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.860090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.97.2:45369)
0929 00:54:15.860664 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.860668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.860682 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:54:15.860751 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.860755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.860829 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.860950 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.861335 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.862096 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.863909 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.864278 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.864367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.864898 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.865405 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.865509 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.865655 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.865715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260929 00:54:15.881804 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.875890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51211 (local address 127.0.97.1:42465)
0929 00:54:15.876105 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.876113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.876681 (+ 568us) server_negotiation.cc:407] Connection header received
0929 00:54:15.876857 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.876860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.876922 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.876994 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.877445 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.878157 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.880147 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.880648 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.880739 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.881232 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.881335 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.881395 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.881540 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.881605 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:54:15.882103 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.875891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42465 (local address 127.0.97.2:51211)
0929 00:54:15.876577 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.876597 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.876729 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.877029 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.877033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.877046 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:15.877319 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.877328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.878283 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.878288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.879928 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.879942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.880809 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.880815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.881001 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.881062 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.881425 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.881435 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.881517 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.881650 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":369,"threads_started":1}
I20260929 00:54:15.891090 2563 raft_consensus.cc:1261] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.891491 2289 raft_consensus.cc:1261] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:15.891724 2684 consensus_queue.cc:1035] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.892195 2639 consensus_queue.cc:1035] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:15.897771 2661 mvcc.cc:205] Tried to move back new op lower bound from 7334474776124305408 to 7334474775884173312. Current Snapshot: MvccSnapshot[applied={T|T < 7334474776124305408}]
I20260929 00:54:15.898896 2660 mvcc.cc:205] Tried to move back new op lower bound from 7334474776124305408 to 7334474775884173312. Current Snapshot: MvccSnapshot[applied={T|T < 7334474776124305408}]
I20260929 00:54:15.921161 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.915196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51297 (local address 127.0.97.2:45369)
0929 00:54:15.915344 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:15.915350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:15.915821 (+ 471us) server_negotiation.cc:407] Connection header received
0929 00:54:15.915917 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:15.915922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:15.916003 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:15.916085 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:15.916419 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.917544 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.919574 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.919971 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.920063 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.920579 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.920676 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.920747 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:15.920905 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 00:54:15.920962 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:54:15.921371 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:15.915236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45369 (local address 127.0.97.1:51297)
0929 00:54:15.915550 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:15.915580 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:15.915701 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:15.916127 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:15.916131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:15.916141 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:54:15.916295 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.916301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.917727 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.917732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.919433 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:15.919448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:15.920120 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:15.920126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:15.920327 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:15.920375 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:54:15.920771 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:54:15.920788 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 00:54:15.920882 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 00:54:15.921033 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 00:54:15.922477 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:54:17.517910 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:17.518121 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:17.521863 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Sending DeleteTablet for 3 replicas of tablet 811ade914daa427a978a410c5d195f6d
I20260929 00:54:17.522559 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 811ade914daa427a978a410c5d195f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41784
I20260929 00:54:17.522572 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 811ade914daa427a978a410c5d195f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260929 00:54:17.522787 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 811ade914daa427a978a410c5d195f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260929 00:54:17.523201 2717 tablet_replica.cc:330] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: stopping tablet replica
I20260929 00:54:17.523442 2717 raft_consensus.cc:2229] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:17.524672 2717 raft_consensus.cc:2258] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.525460 2716 tablet_replica.cc:330] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: stopping tablet replica
I20260929 00:54:17.526782 2717 ts_tablet_manager.cc:1905] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:17.531752 2716 raft_consensus.cc:2229] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:17.532030 2716 raft_consensus.cc:2258] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.533258 2716 ts_tablet_manager.cc:1905] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:17.535085 2717 ts_tablet_manager.cc:1918] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:17.535251 2717 log.cc:1196] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/811ade914daa427a978a410c5d195f6d
I20260929 00:54:17.535758 2717 ts_tablet_manager.cc:1939] T 811ade914daa427a978a410c5d195f6d P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting consensus metadata
I20260929 00:54:17.536129 2718 tablet_replica.cc:330] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: stopping tablet replica
I20260929 00:54:17.536335 2718 raft_consensus.cc:2229] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:17.536688 2718 raft_consensus.cc:2258] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.537194 2115 catalog_manager.cc:4841] TS 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369): tablet 811ade914daa427a978a410c5d195f6d (table test-table [id=511906597a4a475fa659bac971708402]) successfully deleted
I20260929 00:54:17.537693 2718 ts_tablet_manager.cc:1905] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:17.540985 2716 ts_tablet_manager.cc:1918] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:17.541116 2716 log.cc:1196] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/811ade914daa427a978a410c5d195f6d
I20260929 00:54:17.541572 2716 ts_tablet_manager.cc:1939] T 811ade914daa427a978a410c5d195f6d P b7f2396acbc34405a5f863d2c0f87afa: Deleting consensus metadata
I20260929 00:54:17.542687 2113 catalog_manager.cc:4841] TS b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465): tablet 811ade914daa427a978a410c5d195f6d (table test-table [id=511906597a4a475fa659bac971708402]) successfully deleted
I20260929 00:54:17.543610 2718 ts_tablet_manager.cc:1918] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:17.543707 2718 log.cc:1196] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/811ade914daa427a978a410c5d195f6d
I20260929 00:54:17.544196 2718 ts_tablet_manager.cc:1939] T 811ade914daa427a978a410c5d195f6d P dd8e9ddc18b3411b973c11ee35f68440: Deleting consensus metadata
I20260929 00:54:17.545217 2114 catalog_manager.cc:4841] TS dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537): tablet 811ade914daa427a978a410c5d195f6d (table test-table [id=511906597a4a475fa659bac971708402]) successfully deleted
Sep 29 00:54:17 dist-test-slave-pzsx krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:54:17 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:17.573128 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:17.559346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.0.97.62:37819)
0929 00:54:17.559890 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:17.559898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:17.559973 (+ 75us) server_negotiation.cc:407] Connection header received
0929 00:54:17.560197 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:17.560201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:17.560263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:17.560375 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:17.562084 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:17.562976 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:17.564360 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:17.564835 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:17.564975 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:17.568628 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:17.568633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:17.568636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:17.568668 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:17.571001 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:17.571536 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:17.571540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:17.571542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:17.571655 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:17.571941 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:17.571944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:17.571946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:17.572211 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:17.572372 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:17.572725 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 00:54:17.572874 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":256,"threads_started":1}
I20260929 00:54:17.576306 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:17.576892 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:17.583014 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8bc515d34433d85bccbe03885bf72 (DEFAULT_TABLE table=test-table [id=acd344d1e8d347e8b7c4385eefc43823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.583118 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8bc515d34433d85bccbe03885bf72 (DEFAULT_TABLE table=test-table [id=acd344d1e8d347e8b7c4385eefc43823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.583307 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8bc515d34433d85bccbe03885bf72 (DEFAULT_TABLE table=test-table [id=acd344d1e8d347e8b7c4385eefc43823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.583473 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8bc515d34433d85bccbe03885bf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.583534 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8bc515d34433d85bccbe03885bf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.583678 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8bc515d34433d85bccbe03885bf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.590674 2728 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Bootstrap starting.
I20260929 00:54:17.591413 2727 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Bootstrap starting.
I20260929 00:54:17.592522 2729 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Bootstrap starting.
I20260929 00:54:17.593274 2727 tablet_bootstrap.cc:654] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.594095 2728 tablet_bootstrap.cc:654] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.594523 2729 tablet_bootstrap.cc:654] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.595306 2727 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: No bootstrap required, opened a new log
I20260929 00:54:17.595498 2727 ts_tablet_manager.cc:1397] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 00:54:17.596089 2728 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: No bootstrap required, opened a new log
I20260929 00:54:17.596091 2727 raft_consensus.cc:348] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.596262 2728 ts_tablet_manager.cc:1397] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:54:17.596345 2727 raft_consensus.cc:374] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.596438 2727 raft_consensus.cc:729] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.596724 2727 consensus_queue.cc:260] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.597213 2727 ts_tablet_manager.cc:1428] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:54:17.597263 2729 tablet_bootstrap.cc:492] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: No bootstrap required, opened a new log
I20260929 00:54:17.597378 2729 ts_tablet_manager.cc:1397] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 00:54:17.597225 2728 raft_consensus.cc:348] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.597470 2728 raft_consensus.cc:374] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.597559 2728 raft_consensus.cc:729] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.597774 2728 consensus_queue.cc:260] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.598347 2728 ts_tablet_manager.cc:1428] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:54:17.598300 2729 raft_consensus.cc:348] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.598515 2729 raft_consensus.cc:374] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.598606 2729 raft_consensus.cc:729] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8e9ddc18b3411b973c11ee35f68440, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.598839 2729 consensus_queue.cc:260] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.600121 2729 ts_tablet_manager.cc:1428] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:54:17.681056 2693 raft_consensus.cc:482] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:17.681301 2693 raft_consensus.cc:504] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.681968 2693 leader_election.cc:288] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:17.682533 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8bc515d34433d85bccbe03885bf72" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440" is_pre_election: true
I20260929 00:54:17.682533 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8bc515d34433d85bccbe03885bf72" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa" is_pre_election: true
I20260929 00:54:17.682772 2289 raft_consensus.cc:2454] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:17.682772 2563 raft_consensus.cc:2454] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:17.683231 2357 leader_election.cc:302] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:17.683525 2693 raft_consensus.cc:2781] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:17.683662 2693 raft_consensus.cc:482] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:17.683787 2693 raft_consensus.cc:3037] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.685909 2693 raft_consensus.cc:504] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.686452 2693 leader_election.cc:288] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Requested vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:17.686928 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8bc515d34433d85bccbe03885bf72" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440"
I20260929 00:54:17.686906 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8bc515d34433d85bccbe03885bf72" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa"
I20260929 00:54:17.687121 2289 raft_consensus.cc:3037] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.687109 2563 raft_consensus.cc:3037] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.690428 2289 raft_consensus.cc:2454] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:17.690426 2563 raft_consensus.cc:2454] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:17.690836 2357 leader_election.cc:302] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:17.691133 2693 raft_consensus.cc:2781] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:17.691308 2693 raft_consensus.cc:686] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Running, Role: LEADER
I20260929 00:54:17.691571 2693 consensus_queue.cc:237] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.694049 2128 catalog_manager.cc:5495] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:17.723905 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:17.718838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.97.2:45369)
0929 00:54:17.719449 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:17.719455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:17.719472 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:54:17.719516 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:17.719519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:17.719577 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:17.719681 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:17.720046 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:17.720818 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:17.722080 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:17.722645 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:17.722775 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:17.723164 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:17.723261 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:17.723360 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:17.723539 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:54:17.723609 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
I20260929 00:54:17.731854 2563 raft_consensus.cc:1261] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:17.732080 2289 raft_consensus.cc:1261] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:17.732380 2706 consensus_queue.cc:1035] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:17.732683 2693 consensus_queue.cc:1035] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:17.774060 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:17.774302 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:17.776803 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Sending DeleteTablet for 3 replicas of tablet 92c8bc515d34433d85bccbe03885bf72
I20260929 00:54:17.777443 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8bc515d34433d85bccbe03885bf72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41784
I20260929 00:54:17.777549 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8bc515d34433d85bccbe03885bf72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260929 00:54:17.777639 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8bc515d34433d85bccbe03885bf72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260929 00:54:17.777758 2716 tablet_replica.cc:330] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: stopping tablet replica
I20260929 00:54:17.777841 2717 tablet_replica.cc:330] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: stopping tablet replica
I20260929 00:54:17.777943 2716 raft_consensus.cc:2229] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:17.777977 2717 raft_consensus.cc:2229] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:17.778111 2716 raft_consensus.cc:2258] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.778234 2717 raft_consensus.cc:2258] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.778264 2718 tablet_replica.cc:330] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: stopping tablet replica
I20260929 00:54:17.778494 2718 raft_consensus.cc:2229] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:17.778666 2718 raft_consensus.cc:2258] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:17.778724 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 00:54:17.779299 2717 ts_tablet_manager.cc:1905] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 00:54:17.779327 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:17.779394 2716 ts_tablet_manager.cc:1905] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:17.779635 2718 ts_tablet_manager.cc:1905] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:17.783739 2717 ts_tablet_manager.cc:1918] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:17.783836 2717 log.cc:1196] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/92c8bc515d34433d85bccbe03885bf72
I20260929 00:54:17.784193 2717 ts_tablet_manager.cc:1939] T 92c8bc515d34433d85bccbe03885bf72 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting consensus metadata
I20260929 00:54:17.785219 2115 catalog_manager.cc:4841] TS 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369): tablet 92c8bc515d34433d85bccbe03885bf72 (table test-table [id=acd344d1e8d347e8b7c4385eefc43823]) successfully deleted
I20260929 00:54:17.785833 2718 ts_tablet_manager.cc:1918] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:17.785974 2718 log.cc:1196] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/92c8bc515d34433d85bccbe03885bf72
I20260929 00:54:17.786379 2718 ts_tablet_manager.cc:1939] T 92c8bc515d34433d85bccbe03885bf72 P dd8e9ddc18b3411b973c11ee35f68440: Deleting consensus metadata
I20260929 00:54:17.786456 2716 ts_tablet_manager.cc:1918] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:54:17.786582 2716 log.cc:1196] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/92c8bc515d34433d85bccbe03885bf72
I20260929 00:54:17.786985 2716 ts_tablet_manager.cc:1939] T 92c8bc515d34433d85bccbe03885bf72 P b7f2396acbc34405a5f863d2c0f87afa: Deleting consensus metadata
I20260929 00:54:17.787452 2114 catalog_manager.cc:4841] TS dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537): tablet 92c8bc515d34433d85bccbe03885bf72 (table test-table [id=acd344d1e8d347e8b7c4385eefc43823]) successfully deleted
I20260929 00:54:17.788476 2113 catalog_manager.cc:4841] TS b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465): tablet 92c8bc515d34433d85bccbe03885bf72 (table test-table [id=acd344d1e8d347e8b7c4385eefc43823]) successfully deleted
I20260929 00:54:17.790014 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 (DEFAULT_TABLE table=test-table [id=ac8a0d899dd1498db5b7a62861bade19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.790201 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 (DEFAULT_TABLE table=test-table [id=ac8a0d899dd1498db5b7a62861bade19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.790221 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 (DEFAULT_TABLE table=test-table [id=ac8a0d899dd1498db5b7a62861bade19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:17.790416 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dc640da5f14f25b2d37db3efa86a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.790546 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dc640da5f14f25b2d37db3efa86a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.790642 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dc640da5f14f25b2d37db3efa86a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:17.794894 2727 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Bootstrap starting.
I20260929 00:54:17.796139 2727 tablet_bootstrap.cc:654] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.796416 2728 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Bootstrap starting.
I20260929 00:54:17.797464 2729 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Bootstrap starting.
I20260929 00:54:17.797766 2728 tablet_bootstrap.cc:654] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.798633 2727 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: No bootstrap required, opened a new log
I20260929 00:54:17.798789 2727 ts_tablet_manager.cc:1397] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 00:54:17.799436 2729 tablet_bootstrap.cc:654] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:17.799536 2727 raft_consensus.cc:348] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.799790 2727 raft_consensus.cc:374] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.799862 2727 raft_consensus.cc:729] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.800103 2728 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: No bootstrap required, opened a new log
I20260929 00:54:17.800242 2728 ts_tablet_manager.cc:1397] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 00:54:17.800098 2727 consensus_queue.cc:260] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.800989 2728 raft_consensus.cc:348] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.801218 2728 raft_consensus.cc:374] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.801296 2728 raft_consensus.cc:729] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f2396acbc34405a5f863d2c0f87afa, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.801384 2727 ts_tablet_manager.cc:1428] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 00:54:17.801512 2728 consensus_queue.cc:260] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.802008 2728 ts_tablet_manager.cc:1428] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 00:54:17.803467 2729 tablet_bootstrap.cc:492] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: No bootstrap required, opened a new log
I20260929 00:54:17.803647 2729 ts_tablet_manager.cc:1397] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 00:54:17.804961 2729 raft_consensus.cc:348] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.805166 2729 raft_consensus.cc:374] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:17.805253 2729 raft_consensus.cc:729] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8e9ddc18b3411b973c11ee35f68440, State: Initialized, Role: FOLLOWER
I20260929 00:54:17.805469 2729 consensus_queue.cc:260] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.805899 2729 ts_tablet_manager.cc:1428] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 00:54:17.822410 2693 raft_consensus.cc:482] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:17.822638 2693 raft_consensus.cc:504] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.823271 2693 leader_election.cc:288] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:17.823693 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dc640da5f14f25b2d37db3efa86a87" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa" is_pre_election: true
I20260929 00:54:17.823746 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dc640da5f14f25b2d37db3efa86a87" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440" is_pre_election: true
I20260929 00:54:17.824026 2563 raft_consensus.cc:2454] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:17.823987 2287 raft_consensus.cc:2454] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 0.
I20260929 00:54:17.824461 2356 leader_election.cc:302] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, b7f2396acbc34405a5f863d2c0f87afa, dd8e9ddc18b3411b973c11ee35f68440; no voters:
I20260929 00:54:17.824786 2693 raft_consensus.cc:2781] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:17.824910 2693 raft_consensus.cc:482] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:17.825022 2693 raft_consensus.cc:3037] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.827387 2693 raft_consensus.cc:504] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.827873 2693 leader_election.cc:288] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Requested vote from peers b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465), dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537)
I20260929 00:54:17.828287 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dc640da5f14f25b2d37db3efa86a87" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f2396acbc34405a5f863d2c0f87afa"
I20260929 00:54:17.828305 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dc640da5f14f25b2d37db3efa86a87" candidate_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8e9ddc18b3411b973c11ee35f68440"
I20260929 00:54:17.828483 2563 raft_consensus.cc:3037] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.828469 2287 raft_consensus.cc:3037] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:17.830435 2563 raft_consensus.cc:2454] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:17.830413 2287 raft_consensus.cc:2454] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea3dbd3537b4d0d8f9295f7c4ab8646 in term 1.
I20260929 00:54:17.830842 2356 leader_election.cc:302] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea3dbd3537b4d0d8f9295f7c4ab8646, b7f2396acbc34405a5f863d2c0f87afa; no voters:
I20260929 00:54:17.831126 2693 raft_consensus.cc:2781] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:17.831274 2693 raft_consensus.cc:686] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea3dbd3537b4d0d8f9295f7c4ab8646, State: Running, Role: LEADER
I20260929 00:54:17.831542 2693 consensus_queue.cc:237] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } }
I20260929 00:54:17.833864 2128 catalog_manager.cc:5495] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea3dbd3537b4d0d8f9295f7c4ab8646" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:17.857375 2563 raft_consensus.cc:1261] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:17.857546 2287 raft_consensus.cc:1261] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Refusing update from remote peer 0ea3dbd3537b4d0d8f9295f7c4ab8646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:17.857901 2693 consensus_queue.cc:1035] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8e9ddc18b3411b973c11ee35f68440" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:17.858191 2706 consensus_queue.cc:1035] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f2396acbc34405a5f863d2c0f87afa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:54:17.875299 2699 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 00:54:19.451146 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:19.451417 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:19.455343 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 919c9b8de821479899faa13a356e6942: Sending DeleteTablet for 3 replicas of tablet 82dc640da5f14f25b2d37db3efa86a87
I20260929 00:54:19.456476 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48520
I20260929 00:54:19.456466 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41784
I20260929 00:54:19.456667 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dc640da5f14f25b2d37db3efa86a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260929 00:54:19.457120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2200
I20260929 00:54:19.457204 2769 tablet_replica.cc:330] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: stopping tablet replica
I20260929 00:54:19.457463 2769 raft_consensus.cc:2229] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:19.457846 2769 raft_consensus.cc:2258] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:19.462327 2769 ts_tablet_manager.cc:1905] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:19.476852 2771 tablet_replica.cc:330] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: stopping tablet replica
I20260929 00:54:19.477068 2771 raft_consensus.cc:2229] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:19.477404 2771 raft_consensus.cc:2258] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:19.476480 2770 tablet_replica.cc:330] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: stopping tablet replica
I20260929 00:54:19.477837 2770 raft_consensus.cc:2229] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:19.478165 2770 raft_consensus.cc:2258] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:19.478299 2769 ts_tablet_manager.cc:1918] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:19.478469 2769 log.cc:1196] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/82dc640da5f14f25b2d37db3efa86a87
I20260929 00:54:19.478919 2771 ts_tablet_manager.cc:1905] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:19.478941 2769 ts_tablet_manager.cc:1939] T 82dc640da5f14f25b2d37db3efa86a87 P dd8e9ddc18b3411b973c11ee35f68440: Deleting consensus metadata
I20260929 00:54:19.480020 2114 catalog_manager.cc:4841] TS dd8e9ddc18b3411b973c11ee35f68440 (127.0.97.3:36537): tablet 82dc640da5f14f25b2d37db3efa86a87 (table test-table [id=ac8a0d899dd1498db5b7a62861bade19]) successfully deleted
I20260929 00:54:19.480383 2770 ts_tablet_manager.cc:1905] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:19.484972 2771 ts_tablet_manager.cc:1918] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:19.485118 2771 log.cc:1196] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/82dc640da5f14f25b2d37db3efa86a87
I20260929 00:54:19.485445 2770 ts_tablet_manager.cc:1918] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 00:54:19.485529 2771 ts_tablet_manager.cc:1939] T 82dc640da5f14f25b2d37db3efa86a87 P 0ea3dbd3537b4d0d8f9295f7c4ab8646: Deleting consensus metadata
I20260929 00:54:19.485618 2770 log.cc:1196] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/82dc640da5f14f25b2d37db3efa86a87
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:19.486011 2770 ts_tablet_manager.cc:1939] T 82dc640da5f14f25b2d37db3efa86a87 P b7f2396acbc34405a5f863d2c0f87afa: Deleting consensus metadata
I20260929 00:54:19.486927 2113 catalog_manager.cc:4841] TS b7f2396acbc34405a5f863d2c0f87afa (127.0.97.1:42465): tablet 82dc640da5f14f25b2d37db3efa86a87 (table test-table [id=ac8a0d899dd1498db5b7a62861bade19]) successfully deleted
I20260929 00:54:19.486987 2115 catalog_manager.cc:4841] TS 0ea3dbd3537b4d0d8f9295f7c4ab8646 (127.0.97.2:45369): tablet 82dc640da5f14f25b2d37db3efa86a87 (table test-table [id=ac8a0d899dd1498db5b7a62861bade19]) successfully deleted
I20260929 00:54:19.492432 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.458319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.97.1:42465)
0929 00:54:19.473883 (+ 15564us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.473893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.475917 (+ 2024us) server_negotiation.cc:407] Connection header received
0929 00:54:19.476143 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.476151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.476257 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.476387 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.478057 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.479326 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.481084 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.481762 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.481889 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.487348 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.487362 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.487379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.487408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.490217 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.490802 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.490809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.490811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.490924 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.491230 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.491233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.491235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.491569 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.491766 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.491959 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.492138 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15247,"thread_start_us":15310,"threads_started":1}
W20260929 00:54:19.493132 2331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41814
W20260929 00:54:19.493733 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:19.493961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2200
I20260929 00:54:19.524600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2340
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:19.540956 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.525363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.97.2:45369)
0929 00:54:19.525877 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.525881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.525897 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:19.525950 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.525954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.526013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.526104 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.527644 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.528355 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.530231 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.530662 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.530760 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.535654 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.535671 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.535688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.535721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.538558 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.539138 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.539145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.539148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.539269 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.539548 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.539551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.539553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.539883 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.540087 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.540312 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.540615 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
W20260929 00:54:19.541586 2470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60574
W20260929 00:54:19.542192 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:19.542344 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2340
I20260929 00:54:19.563957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2479
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:19.578351 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:19.564691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.0.97.3:36537)
0929 00:54:19.565177 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:19.565182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:19.565197 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:54:19.565250 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:19.565254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:19.565312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:19.565408 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:19.566599 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.567312 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.568436 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:19.568883 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:19.568991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:19.572939 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:19.572954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:19.572971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:19.573001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:19.575995 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.576641 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.576649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.576651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.576805 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:19.577114 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:19.577117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:19.577119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:19.577434 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:19.577654 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:19.577827 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 00:54:19.577980 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
W20260929 00:54:19.579002 2609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48532
W20260929 00:54:19.579591 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:19.579736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2479
I20260929 00:54:19.596632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2097
I20260929 00:54:19.597678 2169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:19.711225 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2097
2026-09-29T00:54:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7865 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2779](info): set up 2 sockets
Sep 29 00:54:19 dist-test-slave-pzsx krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260929 00:54:20.789058 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:20 dist-test-slave-pzsx krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:20.975859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34897
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:21.291149 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:21.291445 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:21.291531 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:21.299352 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:21.299459 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:21.299505 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:21.299561 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:21.299611 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:21.309335 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:21.310587 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:21.311828 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:21.323205 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:21.323244 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:21.323285 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:21.324194 2792 server_base.cc:1034] running on GCE node
I20260929 00:54:21.324996 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:21.326385 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:21.327612 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790643261327557 us; error 62 us; skew 500 ppm
I20260929 00:54:21.327957 2792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:21 dist-test-slave-pzsx krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:21.335078 2792 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:21.336665 2792 webserver.cc:460] Webserver started at http://127.0.97.62:38081/ using document root <none> and password file <none>
I20260929 00:54:21.337363 2792 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:21.337507 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:21.337821 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:21.340785 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "e9c761e51876442b846971e5c1eb2f1d"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "29fe2cad945307f6ee807898faf659f4"
server_key_iv: "619f490a96e7666f85ef4bd0a247ab37"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.341542 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9c761e51876442b846971e5c1eb2f1d"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "29fe2cad945307f6ee807898faf659f4"
server_key_iv: "619f490a96e7666f85ef4bd0a247ab37"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.346565 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:21.350028 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:21.351792 2792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:21.351960 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "e9c761e51876442b846971e5c1eb2f1d"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "29fe2cad945307f6ee807898faf659f4"
server_key_iv: "619f490a96e7666f85ef4bd0a247ab37"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.352219 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:21.377020 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:21.381003 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:21.381294 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:21.398936 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34897
I20260929 00:54:21.398969 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34897 every 8 connection(s)
I20260929 00:54:21.400266 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:21.403851 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:21.404271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2792
I20260929 00:54:21.404534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:21.405187 388 external_mini_cluster.cc:1428] Setting key 03d40687be792ddcc4aa52b2d0dc73de
I20260929 00:54:21.417837 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d: Bootstrap starting.
Sep 29 00:54:21 dist-test-slave-pzsx krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:21.421032 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:21.422351 2859 log.cc:824] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:21.425606 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d: No bootstrap required, opened a new log
I20260929 00:54:21.428448 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:21.407226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.97.62:34897)
0929 00:54:21.407988 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:21.408002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:21.408051 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:54:21.409874 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:21.409899 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:21.410396 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:21.410815 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:21.412153 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.414032 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.415239 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.416280 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.416444 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:21.420363 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:21.420373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:21.420390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:21.420429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:21.424118 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:21.424844 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:21.424852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:21.424860 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:21.425058 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:21.425402 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:21.425406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:21.425409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:21.426061 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:21.426349 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:21.426682 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 00:54:21.427076 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":344,"threads_started":1}
I20260929 00:54:21.434430 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } }
I20260929 00:54:21.434762 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:21.434834 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c761e51876442b846971e5c1eb2f1d, State: Initialized, Role: FOLLOWER
I20260929 00:54:21.435355 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } }
I20260929 00:54:21.435557 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:21.435694 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:21.435878 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:21.438184 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } }
I20260929 00:54:21.438601 2859 leader_election.cc:302] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9c761e51876442b846971e5c1eb2f1d; no voters:
I20260929 00:54:21.439148 2859 leader_election.cc:288] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:21.439289 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:21.439915 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [term 1 LEADER]: Becoming Leader. State: Replica: e9c761e51876442b846971e5c1eb2f1d, State: Running, Role: LEADER
I20260929 00:54:21.440593 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } }
I20260929 00:54:21.441330 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:21.443610 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9c761e51876442b846971e5c1eb2f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } } }
I20260929 00:54:21.443596 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9c761e51876442b846971e5c1eb2f1d. Latest consensus state: current_term: 1 leader_uuid: "e9c761e51876442b846971e5c1eb2f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c761e51876442b846971e5c1eb2f1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34897 } } }
I20260929 00:54:21.444031 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:21.444067 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:21.445631 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:21.450678 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:21.459316 2872 catalog_manager.cc:1347] Generated new cluster ID: 09e11507ca6c427bbe2ee7d12b2a450a
I20260929 00:54:21.459430 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:21.471448 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:21.472775 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:21.479218 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9c761e51876442b846971e5c1eb2f1d: Generated new TSK 0
I20260929 00:54:21.479980 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:21.532549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:21.846983 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:21.847328 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:21.847501 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:21.854813 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:21.854945 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:21.855170 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:21.865063 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:21.866317 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:21.867660 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:21.880266 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:21.880340 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:21.880785 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:21.881188 2885 server_base.cc:1034] running on GCE node
I20260929 00:54:21.881747 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:21.882613 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:21.883832 2885 hybrid_clock.cc:648] HybridClock initialized: now 1790643261883788 us; error 57 us; skew 500 ppm
I20260929 00:54:21.884136 2885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:21 dist-test-slave-pzsx krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:21.887162 2885 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:21.888645 2885 webserver.cc:460] Webserver started at http://127.0.97.1:37807/ using document root <none> and password file <none>
I20260929 00:54:21.889323 2885 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:21.889465 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:21.889846 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:21.893958 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "2e749769c1bc4bfbbfba0caec15f7c4b"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "7391e2056e82afc97f5aa00f6ecc7ca9"
server_key_iv: "63891f3923ed3708ee5d41aae8fa1c2b"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.895017 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2e749769c1bc4bfbbfba0caec15f7c4b"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "7391e2056e82afc97f5aa00f6ecc7ca9"
server_key_iv: "63891f3923ed3708ee5d41aae8fa1c2b"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.902168 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 00:54:21.905800 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:21.907150 2885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:21.907325 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "2e749769c1bc4bfbbfba0caec15f7c4b"
format_stamp: "Formatted at 2026-09-29 00:54:21 on dist-test-slave-pzsx"
server_key: "7391e2056e82afc97f5aa00f6ecc7ca9"
server_key_iv: "63891f3923ed3708ee5d41aae8fa1c2b"
server_key_version: "encryptionkey@0"
I20260929 00:54:21.907521 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:21.929241 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:21.933547 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:21.933868 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:21.934798 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:21.936066 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:21.936172 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:21.936311 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:21.936383 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:21.974392 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35355
I20260929 00:54:21.974484 3012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35355 every 8 connection(s)
I20260929 00:54:21.975706 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:21.980973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 2885
I20260929 00:54:21.981200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:21.981997 388 external_mini_cluster.cc:1428] Setting key 59bbc82f44a885e355708a2544e65683
Sep 29 00:54:21 dist-test-slave-pzsx krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:21.997085 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:21.978171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52871 (local address 127.0.97.62:34897)
0929 00:54:21.978656 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:21.978662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:21.979567 (+ 905us) server_negotiation.cc:407] Connection header received
0929 00:54:21.981725 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:21.981729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:21.981793 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:21.981888 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:21.983663 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.984410 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.986949 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.987337 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.987430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:21.991466 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:21.991470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:21.991474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:21.991501 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:21.993950 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:21.994844 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:21.994848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:21.994850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:21.994955 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:21.995415 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:21.995418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:21.995420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:21.995643 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:21.995745 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:21.996682 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 00:54:21.996837 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":264,"threads_started":1}
I20260929 00:54:21.998670 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:21.978761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34897 (local address 127.0.97.1:52871)
0929 00:54:21.979428 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:21.979465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:21.981424 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:21.982168 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:21.982180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:21.982794 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:21.983486 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:21.983508 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.984586 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.984594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:21.986799 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:21.986817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:21.987551 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:21.987560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:21.987888 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:21.988590 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:21.988636 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:21.991274 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:21.994079 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:21.994090 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:21.994109 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:21.994735 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:21.995055 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:21.995062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:21.995069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:21.995307 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:21.995768 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:21.995779 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:21.996227 (+ 448us) client_negotiation.cc:769] Sending connection context
0929 00:54:21.996556 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 00:54:21.997138 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":249,"threads_started":1}
I20260929 00:54:22.000309 3013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34897
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:22.000660 3013 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:22.001363 3013 heartbeater.cc:511] Master 127.0.97.62:34897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:54:22.003630 2824 ts_manager.cc:194] Registered new tserver with Master: 2e749769c1bc4bfbbfba0caec15f7c4b (127.0.97.1:35355)
I20260929 00:54:22.005908 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:22.027667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:22.341217 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:22.341552 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:22.341646 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:22.348776 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:22.348871 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:22.348999 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:22.359748 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:22.361024 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:22.362277 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:22.374742 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:22.374787 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:22.374972 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:22.375622 3019 server_base.cc:1034] running on GCE node
I20260929 00:54:22.376191 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:22.377161 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:22.378391 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790643262378348 us; error 53 us; skew 500 ppm
I20260929 00:54:22.378696 3019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:22 dist-test-slave-pzsx krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:22.381690 3019 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:22.383101 3019 webserver.cc:460] Webserver started at http://127.0.97.2:45527/ using document root <none> and password file <none>
I20260929 00:54:22.383770 3019 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:22.383912 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:22.384213 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:22.386960 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "76df01b5c55e4bee80190f66ed613012"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "b065f7caaa90723f2a52a25eb7087c86"
server_key_iv: "bf96365f77b46711b3364ad3716336dd"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.387692 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "76df01b5c55e4bee80190f66ed613012"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "b065f7caaa90723f2a52a25eb7087c86"
server_key_iv: "bf96365f77b46711b3364ad3716336dd"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.392423 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 00:54:22.395581 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.396855 3019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:54:22.396973 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "76df01b5c55e4bee80190f66ed613012"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "b065f7caaa90723f2a52a25eb7087c86"
server_key_iv: "bf96365f77b46711b3364ad3716336dd"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.397136 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:22.414860 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:22.418681 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:22.418910 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:22.419747 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:22.421002 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:22.421080 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.421165 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:22.421208 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.452757 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45125
I20260929 00:54:22.452800 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45125 every 8 connection(s)
I20260929 00:54:22.453965 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:22.458662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3019
I20260929 00:54:22.458856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:22.459362 388 external_mini_cluster.cc:1428] Setting key 9a4fdde080ba5815007888749d2256ac
Sep 29 00:54:22 dist-test-slave-pzsx krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:22.476167 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:22.456365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40665 (local address 127.0.97.62:34897)
0929 00:54:22.456620 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:22.456625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:22.457793 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 00:54:22.460064 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:22.460073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:22.460173 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:22.460254 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:22.462115 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.462834 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.465343 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.465737 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.465830 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:22.469364 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:22.469373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:22.469380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:22.469424 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:22.472980 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:22.473659 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:22.473667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:22.473674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:22.473824 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:22.474292 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:22.474299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:22.474304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:22.474677 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:22.474833 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:22.475474 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 00:54:22.475724 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:54:22.476863 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:22.456931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34897 (local address 127.0.97.2:40665)
0929 00:54:22.457599 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:22.457671 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:22.459761 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:22.460538 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:22.460552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:22.461179 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:22.461928 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:22.461952 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.462989 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.462998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:22.465197 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:22.465213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.465956 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.465964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:22.466278 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:22.467060 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:22.467085 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:22.469136 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:22.473105 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:22.473116 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:22.473128 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:22.473527 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:22.473957 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:22.473963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:22.473966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:22.474169 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:22.474794 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:22.474799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:22.475074 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 00:54:22.475284 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 00:54:22.475739 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:22.478287 3147 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34897
I20260929 00:54:22.478605 3147 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:22.479290 3147 heartbeater.cc:511] Master 127.0.97.62:34897 requested a full tablet report, sending...
I20260929 00:54:22.480748 2824 ts_manager.cc:194] Registered new tserver with Master: 76df01b5c55e4bee80190f66ed613012 (127.0.97.2:45125)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:54:22.481719 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:22.503052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:22.811168 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:22.811463 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:22.811637 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:22.818596 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:22.818742 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:22.818907 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:22.828245 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:22.829512 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:22.830814 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:22.843549 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:22.843605 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:22.843631 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:22.844377 3152 server_base.cc:1034] running on GCE node
I20260929 00:54:22.844971 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:22.845806 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:22.846994 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790643262846925 us; error 68 us; skew 500 ppm
I20260929 00:54:22.847254 3152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:22 dist-test-slave-pzsx krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:22.850245 3152 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:22.851626 3152 webserver.cc:460] Webserver started at http://127.0.97.3:33715/ using document root <none> and password file <none>
I20260929 00:54:22.852293 3152 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:22.852437 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:22.852759 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:22.855528 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "07e834fc9b9f439bb7aade5d57169804"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "f3940e92ecf7a8fbeb4d1213eb731ff2"
server_key_iv: "2cedee38789992af3d82c8a479523801"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.856299 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "07e834fc9b9f439bb7aade5d57169804"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "f3940e92ecf7a8fbeb4d1213eb731ff2"
server_key_iv: "2cedee38789992af3d82c8a479523801"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.861173 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:54:22.864387 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.865622 3152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:54:22.865794 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "07e834fc9b9f439bb7aade5d57169804"
format_stamp: "Formatted at 2026-09-29 00:54:22 on dist-test-slave-pzsx"
server_key: "f3940e92ecf7a8fbeb4d1213eb731ff2"
server_key_iv: "2cedee38789992af3d82c8a479523801"
server_key_version: "encryptionkey@0"
I20260929 00:54:22.865991 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:22.880543 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:22.884007 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:22.884313 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:22.885223 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:22.886447 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:22.886546 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.886698 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:22.886773 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:22.918490 3152 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44893
I20260929 00:54:22.918541 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44893 every 8 connection(s)
I20260929 00:54:22.919718 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:22.929441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3152
I20260929 00:54:22.929633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:22.930145 388 external_mini_cluster.cc:1428] Setting key d9be24b8c6dd82d1c1673839c15935d8
Sep 29 00:54:22 dist-test-slave-pzsx krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:22.938585 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:22.922191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59387 (local address 127.0.97.62:34897)
0929 00:54:22.922373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:22.922378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:22.923490 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 00:54:22.925312 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:22.925318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:22.925405 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:22.925510 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:22.926906 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.927665 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.929351 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.929844 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.929945 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:22.933845 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:22.933849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:22.933852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:22.933882 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:22.936076 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:22.936781 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:22.936785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:22.936787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:22.936891 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:22.937243 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:22.937246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:22.937247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:22.937507 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:22.937610 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:22.938207 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 00:54:22.938358 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 00:54:22.939460 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:22.922696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34897 (local address 127.0.97.3:59387)
0929 00:54:22.923346 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:22.923386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:22.925059 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:22.925737 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:22.925744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:22.926211 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:22.926748 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:22.926764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.927822 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.927826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:22.929207 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:22.929223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:22.930067 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:22.930074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:22.930362 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:22.931109 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:22.931130 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:22.933680 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:22.936206 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:22.936212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:22.936224 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:22.936681 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:22.936979 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:22.936982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:22.936984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:22.937151 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:22.937603 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:22.937609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:22.937889 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 00:54:22.938106 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 00:54:22.938450 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":269,"threads_started":1}
I20260929 00:54:22.940730 3280 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34897
I20260929 00:54:22.941007 3280 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:22.941694 3280 heartbeater.cc:511] Master 127.0.97.62:34897 requested a full tablet report, sending...
I20260929 00:54:22.942986 2824 ts_manager.cc:194] Registered new tserver with Master: 07e834fc9b9f439bb7aade5d57169804 (127.0.97.3:44893)
I20260929 00:54:22.943869 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59387
I20260929 00:54:22.945356 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:54:22.973786 3015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:22.971126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.97.62:34897)
0929 00:54:22.971317 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:22.971325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:22.971740 (+ 415us) server_negotiation.cc:407] Connection header received
0929 00:54:22.971983 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:22.971989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:22.972084 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:22.972218 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:22.973443 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40108: BlockingRecv error: recv got EOF from 127.0.0.1:40108 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:54:22.976585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2885
W20260929 00:54:22.979385 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2885: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35355: server requires authentication, but client does not have Kerberos credentials available
I20260929 00:54:22.979557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2885
I20260929 00:54:22.995378 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3019
W20260929 00:54:22.997936 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3019: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45125: server requires authentication, but client does not have Kerberos credentials available
I20260929 00:54:22.998093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3019
I20260929 00:54:23.013470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3152
W20260929 00:54:23.016111 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3152: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44893: server requires authentication, but client does not have Kerberos credentials available
I20260929 00:54:23.016269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3152
I20260929 00:54:23.031388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 2792
I20260929 00:54:23.032557 2854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:23.133474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 2792
2026-09-29T00:54:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3419 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:23 dist-test-slave-pzsx krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:23 dist-test-slave-pzsx krb5kdc[3297](info): set up 2 sockets
Sep 29 00:54:23 dist-test-slave-pzsx krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260929 00:54:24.220944 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:24 dist-test-slave-pzsx krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:24.404815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46065
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:24.713104 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:24.713385 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:24.713572 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:24.720726 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:24.720854 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:24.720963 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:24.721056 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:24.721136 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:24.730798 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:24.732053 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:24.733366 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:24.744576 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:24.744616 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:24.744553 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:24.746035 3310 server_base.cc:1034] running on GCE node
I20260929 00:54:24.746753 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:24.748008 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:24.749223 3310 hybrid_clock.cc:648] HybridClock initialized: now 1790643264749185 us; error 54 us; skew 500 ppm
I20260929 00:54:24.749554 3310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:24 dist-test-slave-pzsx krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:24.752556 3310 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:24.753923 3310 webserver.cc:460] Webserver started at http://127.0.97.62:36223/ using document root <none> and password file <none>
I20260929 00:54:24.754572 3310 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:24.754745 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:24.755028 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:24.758037 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "4b7b781d681843b2ad14ddf8674a1a26"
format_stamp: "Formatted at 2026-09-29 00:54:24 on dist-test-slave-pzsx"
server_key: "8aa2f08f8baa820a3bc24e8adfd99da2"
server_key_iv: "4555b145a5ad6b27f19f0c37ca36485d"
server_key_version: "encryptionkey@0"
I20260929 00:54:24.758800 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b7b781d681843b2ad14ddf8674a1a26"
format_stamp: "Formatted at 2026-09-29 00:54:24 on dist-test-slave-pzsx"
server_key: "8aa2f08f8baa820a3bc24e8adfd99da2"
server_key_iv: "4555b145a5ad6b27f19f0c37ca36485d"
server_key_version: "encryptionkey@0"
I20260929 00:54:24.763700 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 00:54:24.766959 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:24.768184 3310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:54:24.768354 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "4b7b781d681843b2ad14ddf8674a1a26"
format_stamp: "Formatted at 2026-09-29 00:54:24 on dist-test-slave-pzsx"
server_key: "8aa2f08f8baa820a3bc24e8adfd99da2"
server_key_iv: "4555b145a5ad6b27f19f0c37ca36485d"
server_key_version: "encryptionkey@0"
I20260929 00:54:24.768556 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:24.784301 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:24.788229 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:24.788515 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:24.806459 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46065
I20260929 00:54:24.806481 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46065 every 8 connection(s)
I20260929 00:54:24.807770 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:24.811324 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:24.812856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3310
I20260929 00:54:24.813071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:24.813839 388 external_mini_cluster.cc:1428] Setting key a088daa5a180a82011e864a0f5f3b788
I20260929 00:54:24.825358 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26: Bootstrap starting.
Sep 29 00:54:24 dist-test-slave-pzsx krb5kdc[3297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:24.828987 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:24.830003 3377 log.cc:824] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:24.832664 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26: No bootstrap required, opened a new log
I20260929 00:54:24.835711 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:24.815623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.97.62:46065)
0929 00:54:24.816388 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:24.816399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:24.816440 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:54:24.818023 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:24.818048 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:24.818426 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:24.818755 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:24.820480 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:24.821767 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:24.823540 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:24.824170 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:24.824289 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:24.828826 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:24.828834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:24.828847 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:24.828888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:24.831911 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:24.832567 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:24.832582 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:24.832590 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:24.832778 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:24.833183 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:24.833186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:24.833188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:24.833667 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:24.833894 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:24.834338 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 00:54:24.834687 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":375,"threads_started":1}
I20260929 00:54:24.839339 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:54:24.841039 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260929 00:54:24.841403 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:24.841514 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b781d681843b2ad14ddf8674a1a26, State: Initialized, Role: FOLLOWER
I20260929 00:54:24.842087 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260929 00:54:24.842329 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:24.842468 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:24.842641 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:24.844763 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260929 00:54:24.845229 3377 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b781d681843b2ad14ddf8674a1a26; no voters:
I20260929 00:54:24.845840 3377 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:24.845992 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:24.847321 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b781d681843b2ad14ddf8674a1a26, State: Running, Role: LEADER
I20260929 00:54:24.847834 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260929 00:54:24.847975 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:24.850353 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7b781d681843b2ad14ddf8674a1a26. Latest consensus state: current_term: 1 leader_uuid: "4b7b781d681843b2ad14ddf8674a1a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } } }
I20260929 00:54:24.850675 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:24.851200 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7b781d681843b2ad14ddf8674a1a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b781d681843b2ad14ddf8674a1a26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } } }
I20260929 00:54:24.851507 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:24.853281 3388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:24.860069 3388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:24.877950 3388 catalog_manager.cc:1347] Generated new cluster ID: 74b975b9c30b47e48c292053b91c9f75
I20260929 00:54:24.878072 3388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:24.907047 3388 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:24.909685 3388 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:24.922328 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:24.906433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.0.97.62:46065)
0929 00:54:24.906658 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:24.906667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:24.913173 (+ 6506us) server_negotiation.cc:407] Connection header received
0929 00:54:24.914299 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:24.914306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:24.914380 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:24.914493 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:24.915801 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:24.916929 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:24.918740 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:24.919375 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:24.919477 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:24.920718 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:24.920725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:24.920729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:24.921554 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:24.921623 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:24.921651 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:24.921747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:24.921900 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 00:54:24.921980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:54:24.926069 3388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7b781d681843b2ad14ddf8674a1a26: Generated new TSK 0
I20260929 00:54:24.926860 3388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:54:24.935053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3310
I20260929 00:54:24.936035 3372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:25.023365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3310
2026-09-29T00:54:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1884 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:25 dist-test-slave-pzsx krb5kdc[3412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:25 dist-test-slave-pzsx krb5kdc[3412](info): set up 2 sockets
Sep 29 00:54:25 dist-test-slave-pzsx krb5kdc[3412](info): commencing operation
krb5kdc: starting...
W20260929 00:54:26.088805 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:26 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:26.270149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42363
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:26.588364 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:26.588723 3425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:26.588891 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:26.595973 3425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:26.596132 3425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:26.596241 3425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:26.596345 3425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:26.596410 3425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:26.605890 3425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:26.607234 3425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:26.608580 3425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:26.620359 3430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:26.620417 3431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:26.620355 3433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:26.621356 3425 server_base.cc:1034] running on GCE node
I20260929 00:54:26.622045 3425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:26.623340 3425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:26.624580 3425 hybrid_clock.cc:648] HybridClock initialized: now 1790643266624537 us; error 35 us; skew 500 ppm
I20260929 00:54:26.624927 3425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:26 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:26.637126 3425 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:26.638700 3425 webserver.cc:460] Webserver started at http://127.0.97.62:35945/ using document root <none> and password file <none>
I20260929 00:54:26.639412 3425 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:26.639561 3425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:26.639873 3425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:26.642884 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41"
format_stamp: "Formatted at 2026-09-29 00:54:26 on dist-test-slave-pzsx"
server_key: "052adb867105615f020763390ef89149"
server_key_iv: "cb8dd1b6c3a7f56616ca49ce4c9a345f"
server_key_version: "encryptionkey@0"
I20260929 00:54:26.643653 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41"
format_stamp: "Formatted at 2026-09-29 00:54:26 on dist-test-slave-pzsx"
server_key: "052adb867105615f020763390ef89149"
server_key_iv: "cb8dd1b6c3a7f56616ca49ce4c9a345f"
server_key_version: "encryptionkey@0"
I20260929 00:54:26.648757 3425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 00:54:26.652050 3440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:26.653393 3425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:26.653575 3425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41"
format_stamp: "Formatted at 2026-09-29 00:54:26 on dist-test-slave-pzsx"
server_key: "052adb867105615f020763390ef89149"
server_key_iv: "cb8dd1b6c3a7f56616ca49ce4c9a345f"
server_key_version: "encryptionkey@0"
I20260929 00:54:26.653771 3425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:26.665124 3425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:26.669494 3425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:26.669796 3425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:26.687832 3425 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42363
I20260929 00:54:26.687855 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42363 every 8 connection(s)
I20260929 00:54:26.689333 3425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:26.692888 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:26.697005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3425
I20260929 00:54:26.697232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:26.697840 388 external_mini_cluster.cc:1428] Setting key 2f00f1ac5b2f4b75282d491324d2bb63
I20260929 00:54:26.703284 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41: Bootstrap starting.
I20260929 00:54:26.706146 3492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:26.707058 3492 log.cc:824] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:26.710353 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41: No bootstrap required, opened a new log
Sep 29 00:54:26 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:26.718267 3492 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } }
I20260929 00:54:26.718683 3492 raft_consensus.cc:374] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:26.718819 3492 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12ee9f399db4ed4a6b7a2ed971f4d41, State: Initialized, Role: FOLLOWER
I20260929 00:54:26.719395 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } }
I20260929 00:54:26.719615 3492 raft_consensus.cc:388] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:26.719771 3492 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:26.719947 3492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:26.720072 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:26.699827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.0.97.62:42363)
0929 00:54:26.700647 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:26.700659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:26.700697 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:54:26.702302 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:26.702323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:26.702691 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:26.703074 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:26.704392 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:26.705721 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:26.706980 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:26.707956 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:26.708158 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:26.712618 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:26.712629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:26.712643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:26.712676 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:26.715663 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:26.716325 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:26.716337 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:26.716349 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:26.716589 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:26.716971 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:26.716976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:26.716980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:26.717506 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:26.717818 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:26.718231 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 00:54:26.718705 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260929 00:54:26.722388 3492 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } }
I20260929 00:54:26.722934 3492 leader_election.cc:302] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12ee9f399db4ed4a6b7a2ed971f4d41; no voters:
I20260929 00:54:26.723520 3492 leader_election.cc:288] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:26.723656 3497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:26.724313 3497 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [term 1 LEADER]: Becoming Leader. State: Replica: e12ee9f399db4ed4a6b7a2ed971f4d41, State: Running, Role: LEADER
I20260929 00:54:26.724817 3497 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } }
I20260929 00:54:26.725452 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:26.727625 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } } }
I20260929 00:54:26.728041 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:26.728868 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12ee9f399db4ed4a6b7a2ed971f4d41. Latest consensus state: current_term: 1 leader_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12ee9f399db4ed4a6b7a2ed971f4d41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42363 } } }
I20260929 00:54:26.729478 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:26.729744 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:26.734814 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:26.743665 3504 catalog_manager.cc:1347] Generated new cluster ID: 11473ea8e49646289e6fc65c54835cd3
I20260929 00:54:26.743801 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:26.760003 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:26.761619 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:26.772490 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12ee9f399db4ed4a6b7a2ed971f4d41: Generated new TSK 0
I20260929 00:54:26.773301 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:26.828756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:27.159080 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:27.159350 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:27.159441 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:27.166623 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:27.166741 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:27.166874 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:27.176674 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:27.177863 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:27.179190 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:27.191699 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:27.191819 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:27.191756 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:27.192371 3518 server_base.cc:1034] running on GCE node
I20260929 00:54:27.193259 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:27.194104 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:27.195326 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790643267195280 us; error 64 us; skew 500 ppm
I20260929 00:54:27.195632 3518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:27 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:27.198647 3518 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:27.199990 3518 webserver.cc:460] Webserver started at http://127.0.97.1:34461/ using document root <none> and password file <none>
I20260929 00:54:27.200734 3518 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:27.200861 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:27.201156 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:27.204026 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "694db48555a94a9ba57c7ae2e02d11ac"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "d691ea9ee48cf8fa3e58e2278f40e9d5"
server_key_iv: "8c044f8e8e2b84aaca6f5f64df1a8215"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.204886 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "694db48555a94a9ba57c7ae2e02d11ac"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "d691ea9ee48cf8fa3e58e2278f40e9d5"
server_key_iv: "8c044f8e8e2b84aaca6f5f64df1a8215"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.210049 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 00:54:27.213439 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.214720 3518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 00:54:27.214874 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "694db48555a94a9ba57c7ae2e02d11ac"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "d691ea9ee48cf8fa3e58e2278f40e9d5"
server_key_iv: "8c044f8e8e2b84aaca6f5f64df1a8215"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.215094 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:27.226548 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:27.230516 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:27.230813 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:27.231805 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:27.233141 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:27.233209 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.233361 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:27.233417 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.265710 3518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34569
I20260929 00:54:27.265796 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34569 every 8 connection(s)
I20260929 00:54:27.266938 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:27.270308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3518
I20260929 00:54:27.270505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:27.271164 388 external_mini_cluster.cc:1428] Setting key fcbbc0b4cea6d2d01472c80da56ac3ff
Sep 29 00:54:27 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:27.287295 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:27.269464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35051 (local address 127.0.97.62:42363)
0929 00:54:27.269882 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:27.269887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:27.270834 (+ 947us) server_negotiation.cc:407] Connection header received
0929 00:54:27.273070 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:27.273076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:27.273153 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:27.273277 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:27.274768 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.275477 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.277179 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.277572 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.277707 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:27.281588 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:27.281592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:27.281595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:27.281624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:27.284025 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:27.285028 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:27.285032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:27.285035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:27.285139 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:27.285629 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:27.285632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:27.285634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:27.285864 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:27.285964 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:27.286878 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 00:54:27.287032 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":233,"threads_started":1}
I20260929 00:54:27.288906 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:27.269884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42363 (local address 127.0.97.1:35051)
0929 00:54:27.270668 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:27.270719 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:27.272763 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:27.273511 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:27.273520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:27.274016 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:27.274588 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:27.274604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.275645 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.275650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:27.277024 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:27.277040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.277795 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.277802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:27.278027 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:27.278704 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:27.278725 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:27.281363 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:27.284182 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:27.284194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:27.284214 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:27.284900 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:27.285252 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:27.285260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:27.285266 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:27.285516 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:27.285988 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:27.285998 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:27.286469 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 00:54:27.286799 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 00:54:27.287371 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":354,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:27.290427 3646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42363
I20260929 00:54:27.290812 3646 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:27.291781 3646 heartbeater.cc:511] Master 127.0.97.62:42363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:54:27.295243 3457 ts_manager.cc:194] Registered new tserver with Master: 694db48555a94a9ba57c7ae2e02d11ac (127.0.97.1:34569)
I20260929 00:54:27.296772 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:27.315723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:27.631541 3652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:27.631821 3652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:27.632006 3652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:27.639366 3652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:27.639520 3652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:27.639709 3652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:27.649598 3652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:27.650938 3652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:27.652272 3652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:27.665307 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:27.665395 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:27.665341 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:27.666543 3652 server_base.cc:1034] running on GCE node
I20260929 00:54:27.667255 3652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:27.668208 3652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:27.669433 3652 hybrid_clock.cc:648] HybridClock initialized: now 1790643267669382 us; error 46 us; skew 500 ppm
I20260929 00:54:27.669768 3652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:27 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:27.672947 3652 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:27.674402 3652 webserver.cc:460] Webserver started at http://127.0.97.2:43513/ using document root <none> and password file <none>
I20260929 00:54:27.675045 3652 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:27.675148 3652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:27.675393 3652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:27.678277 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "e36840aa74054bcfa134234a0343e16a"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "3e71774c6ca1f8f9f6a127b8977a63dd"
server_key_iv: "bc714ba1b3731353514fee1aa0bdfd13"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.678995 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e36840aa74054bcfa134234a0343e16a"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "3e71774c6ca1f8f9f6a127b8977a63dd"
server_key_iv: "bc714ba1b3731353514fee1aa0bdfd13"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.683923 3652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:54:27.687400 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.689046 3652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 00:54:27.689224 3652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "e36840aa74054bcfa134234a0343e16a"
format_stamp: "Formatted at 2026-09-29 00:54:27 on dist-test-slave-pzsx"
server_key: "3e71774c6ca1f8f9f6a127b8977a63dd"
server_key_iv: "bc714ba1b3731353514fee1aa0bdfd13"
server_key_version: "encryptionkey@0"
I20260929 00:54:27.689435 3652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:27.704813 3652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:27.709955 3652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:27.710235 3652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:27.711126 3652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:27.712386 3652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:27.712456 3652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.712626 3652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:27.712707 3652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:27.745878 3652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42439
I20260929 00:54:27.745920 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42439 every 8 connection(s)
I20260929 00:54:27.747179 3652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:27.753262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3652
I20260929 00:54:27.753436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:27.753858 388 external_mini_cluster.cc:1428] Setting key 145b5d66468bd2d3dc8b0d92bd5049f7
Sep 29 00:54:27 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:27.769948 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:27.749746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46451 (local address 127.0.97.62:42363)
0929 00:54:27.749928 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:27.749932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:27.751020 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 00:54:27.752837 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:27.752843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:27.752949 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:27.753060 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:27.754681 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.755756 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.758268 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.758667 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.758828 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:27.762984 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:27.762994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:27.763002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:27.763046 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:27.766631 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:27.767343 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:27.767351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:27.767357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:27.767505 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:27.767912 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:27.767920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:27.767925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:27.768311 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:27.768458 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:27.769244 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 00:54:27.769502 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:54:27.770754 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:27.750190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42363 (local address 127.0.97.2:46451)
0929 00:54:27.750867 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:27.750904 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:27.752578 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:27.753294 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:27.753302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:27.753883 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:27.754485 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:27.754502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.755905 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.755913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:27.758136 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:27.758153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:27.758947 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:27.758956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:27.759274 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:27.760067 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:27.760101 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:27.762750 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:27.766773 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:27.766780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:27.766794 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:27.767203 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:27.767610 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:27.767613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:27.767615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:27.767792 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:27.768425 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:27.768431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:27.768847 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 00:54:27.769089 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 00:54:27.769476 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260929 00:54:27.772324 3780 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42363
I20260929 00:54:27.772730 3780 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:27.773433 3780 heartbeater.cc:511] Master 127.0.97.62:42363 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:27.774917 3457 ts_manager.cc:194] Registered new tserver with Master: e36840aa74054bcfa134234a0343e16a (127.0.97.2:42439)
I20260929 00:54:27.775717 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:27.800954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:28.124616 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:28.124957 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:28.125177 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:28.132452 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:28.132642 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:28.132835 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:28.142578 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:28.143971 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:28.145342 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:28.158756 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:28.158797 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:28.158846 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:28.159943 3785 server_base.cc:1034] running on GCE node
I20260929 00:54:28.160696 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:28.161679 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:28.162916 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790643268162851 us; error 61 us; skew 500 ppm
I20260929 00:54:28.163261 3785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:28.166502 3785 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:28.167933 3785 webserver.cc:460] Webserver started at http://127.0.97.3:38771/ using document root <none> and password file <none>
I20260929 00:54:28.168685 3785 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:28.168785 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:28.169062 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:28.171856 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "d2e8b31db6fe48f8b3a6e5356be59992"
format_stamp: "Formatted at 2026-09-29 00:54:28 on dist-test-slave-pzsx"
server_key: "ff96902e679194738ea8b0c83d4d0d5c"
server_key_iv: "ac402fd3f7fd14cc0a575eb8a9bb6ea9"
server_key_version: "encryptionkey@0"
I20260929 00:54:28.172649 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e8b31db6fe48f8b3a6e5356be59992"
format_stamp: "Formatted at 2026-09-29 00:54:28 on dist-test-slave-pzsx"
server_key: "ff96902e679194738ea8b0c83d4d0d5c"
server_key_iv: "ac402fd3f7fd14cc0a575eb8a9bb6ea9"
server_key_version: "encryptionkey@0"
I20260929 00:54:28.177729 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:28.181295 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:28.182655 3785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:28.182788 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "d2e8b31db6fe48f8b3a6e5356be59992"
format_stamp: "Formatted at 2026-09-29 00:54:28 on dist-test-slave-pzsx"
server_key: "ff96902e679194738ea8b0c83d4d0d5c"
server_key_iv: "ac402fd3f7fd14cc0a575eb8a9bb6ea9"
server_key_version: "encryptionkey@0"
I20260929 00:54:28.182953 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:28.205854 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:28.210479 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:28.210743 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:28.211668 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:28.213001 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:28.213080 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:28.213165 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:28.213208 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:28.245002 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37897
I20260929 00:54:28.245051 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37897 every 8 connection(s)
I20260929 00:54:28.246341 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:28.249733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3785
I20260929 00:54:28.249914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:28.250540 388 external_mini_cluster.cc:1428] Setting key d5bcba044dbbbe59a4829ae217672776
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:28.267995 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.249104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52095 (local address 127.0.97.62:42363)
0929 00:54:28.249281 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:28.249286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:28.250773 (+ 1487us) server_negotiation.cc:407] Connection header received
0929 00:54:28.252923 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:28.252928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:28.253034 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:28.253150 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:28.255272 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.256016 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.257753 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.258191 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.258330 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.261986 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:28.261991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:28.261995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:28.262028 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:28.264478 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.265484 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.265489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.265496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.265632 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.266187 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.266192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.266197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.266569 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:28.266724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:28.267425 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 00:54:28.267658 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:54:28.268982 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.249765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42363 (local address 127.0.97.3:52095)
0929 00:54:28.250565 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:28.250608 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:28.252657 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:28.253445 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:28.253458 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:28.254239 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:28.255046 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:28.255066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.256177 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.256182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:28.257596 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:28.257614 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.258420 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.258428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:28.258729 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.259404 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:28.259428 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:28.261748 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:28.264701 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:28.264711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:28.264730 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:28.265348 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:28.265750 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:28.265755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:28.265759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:28.266039 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:28.266699 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:28.266705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:28.267042 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 00:54:28.267271 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 00:54:28.267672 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":331,"threads_started":1}
I20260929 00:54:28.270323 3913 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42363
I20260929 00:54:28.270622 3913 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:28.271343 3913 heartbeater.cc:511] Master 127.0.97.62:42363 requested a full tablet report, sending...
I20260929 00:54:28.272782 3457 ts_manager.cc:194] Registered new tserver with Master: d2e8b31db6fe48f8b3a6e5356be59992 (127.0.97.3:37897)
I20260929 00:54:28.273561 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52095
I20260929 00:54:28.278139 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:28.299829 3646 heartbeater.cc:503] Master 127.0.97.62:42363 was elected leader, sending a full tablet report...
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:28.325012 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.311591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.97.62:42363)
0929 00:54:28.311844 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:28.311853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:28.312241 (+ 388us) server_negotiation.cc:407] Connection header received
0929 00:54:28.312386 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:28.312390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:28.312450 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:28.312600 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:28.313893 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.314720 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.315881 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.316261 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.316349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.320212 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:28.320217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:28.320220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:28.320253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:28.322695 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.323311 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.323315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.323318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.323430 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.323753 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.323756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.323758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.324043 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:28.324228 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:28.324475 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 00:54:28.324696 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:54:28.325744 3457 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36512
I20260929 00:54:28.327342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3518
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:28.343065 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.328467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.97.1:34569)
0929 00:54:28.329147 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:28.329154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:28.329178 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:54:28.329232 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:28.329238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:28.329445 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:28.329566 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:28.330759 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.331863 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.333079 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.333628 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.333723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.337786 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:28.337799 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:28.337815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:28.337852 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:28.340491 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.341234 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.341241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.341243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.341367 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.341689 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.341692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.341694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.342016 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:28.342242 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:28.342474 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 00:54:28.342670 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":361,"threads_started":1}
W20260929 00:54:28.344405 3641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50484
W20260929 00:54:28.345597 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:28.345803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3518
I20260929 00:54:28.361629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3652
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:28.377681 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.362570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.97.2:42439)
0929 00:54:28.363066 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:28.363073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:28.363098 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:54:28.363156 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:28.363162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:28.363390 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:28.363577 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:28.364925 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.365882 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.367130 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.367675 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.367820 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.371989 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:28.372002 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:28.372018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:28.372067 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:28.374776 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.375398 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.375405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.375407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.375558 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.375901 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.375905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.375907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.376269 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:28.376589 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:28.376871 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 00:54:28.377206 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
W20260929 00:54:28.379354 3775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53362
W20260929 00:54:28.380890 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:28.381101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3652
I20260929 00:54:28.396373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3785
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643268, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:28.412851 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:28.397640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.0.97.3:37897)
0929 00:54:28.398229 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:28.398236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:28.398264 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:28.398321 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:28.398328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:28.398531 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:28.398725 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:28.400049 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.401161 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.402372 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:28.402922 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:28.403057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:28.407493 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:28.407504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:28.407520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:28.407555 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:28.410177 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.410785 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.410792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.410794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.410919 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:28.411246 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:28.411248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:28.411250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:28.411559 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:28.411780 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:28.412315 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 00:54:28.412552 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":316,"threads_started":1}
W20260929 00:54:28.414131 3908 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58722
W20260929 00:54:28.415267 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:54:28.415445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3785
I20260929 00:54:28.430773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3425
I20260929 00:54:28.431965 3487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:28.536011 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3425
2026-09-29T00:54:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3518 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3930](info): set up 2 sockets
Sep 29 00:54:28 dist-test-slave-pzsx krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260929 00:54:29.633018 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:29 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:29.824009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:30.139626 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:30.139988 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:30.140158 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:30.147372 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:30.147545 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:30.147660 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:30.147763 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:30.147838 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:30.157857 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:30.159195 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:30.160566 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:30.172142 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:30.172118 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:30.172094 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:30.173336 3943 server_base.cc:1034] running on GCE node
I20260929 00:54:30.174130 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:30.175568 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:30.176828 3943 hybrid_clock.cc:648] HybridClock initialized: now 1790643270176791 us; error 63 us; skew 500 ppm
I20260929 00:54:30.177160 3943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:30 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:30.180171 3943 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:30.181679 3943 webserver.cc:460] Webserver started at http://127.0.97.62:46173/ using document root <none> and password file <none>
I20260929 00:54:30.182346 3943 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:30.182483 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:30.182801 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:30.185781 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "7d5e70413e5747b3bb06574770844127"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "d7fdeef7514c0f7f6f11d74eeb7cc24e"
server_key_iv: "2136ad8323a2b3bada50eca90e8c6bec"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.186568 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d5e70413e5747b3bb06574770844127"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "d7fdeef7514c0f7f6f11d74eeb7cc24e"
server_key_iv: "2136ad8323a2b3bada50eca90e8c6bec"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.191603 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 00:54:30.194896 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:30.196246 3943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:30.196420 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "7d5e70413e5747b3bb06574770844127"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "d7fdeef7514c0f7f6f11d74eeb7cc24e"
server_key_iv: "2136ad8323a2b3bada50eca90e8c6bec"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.196662 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:30.209826 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:30.213620 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:30.213891 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:30.232134 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42293
I20260929 00:54:30.232156 4009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42293 every 8 connection(s)
I20260929 00:54:30.233588 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:30.237353 4010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:30.241677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 3943
I20260929 00:54:30.241916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:30.242517 388 external_mini_cluster.cc:1428] Setting key fdd7c4dd7b662555453bfd64c156e864
I20260929 00:54:30.248185 4010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127: Bootstrap starting.
I20260929 00:54:30.251870 4010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:30.253047 4010 log.cc:824] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:30.255744 4010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127: No bootstrap required, opened a new log
Sep 29 00:54:30 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:30.263808 4010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260929 00:54:30.264266 4010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:30.264386 4010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5e70413e5747b3bb06574770844127, State: Initialized, Role: FOLLOWER
I20260929 00:54:30.265013 4010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260929 00:54:30.265261 4010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:30.265287 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:30.244381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.0.97.62:42293)
0929 00:54:30.245223 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:30.245235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:30.245271 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:30.246986 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:30.247007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:30.247386 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:30.247708 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:30.249570 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.250858 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.252636 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.253364 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.253511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:30.257675 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:30.257687 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:30.257706 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:30.257760 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:30.261163 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:30.261825 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:30.261834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:30.261847 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:30.262000 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:30.262410 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:30.262415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:30.262426 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:30.262962 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:30.263271 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:30.263709 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 00:54:30.264105 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":375,"threads_started":1}
I20260929 00:54:30.265430 4010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:30.265648 4010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:30.268208 4010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260929 00:54:30.268699 4010 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5e70413e5747b3bb06574770844127; no voters:
I20260929 00:54:30.269263 4010 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:30.269407 4015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:30.270057 4015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5e70413e5747b3bb06574770844127, State: Running, Role: LEADER
I20260929 00:54:30.270619 4015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } }
I20260929 00:54:30.271463 4010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:30.273782 4016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5e70413e5747b3bb06574770844127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } } }
I20260929 00:54:30.274374 4016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:30.273952 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5e70413e5747b3bb06574770844127. Latest consensus state: current_term: 1 leader_uuid: "7d5e70413e5747b3bb06574770844127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e70413e5747b3bb06574770844127" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42293 } } }
I20260929 00:54:30.274955 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:30.277490 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:30.282325 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:30.291460 4024 catalog_manager.cc:1347] Generated new cluster ID: 29a2355596864312b3a84502f1cedc0a
I20260929 00:54:30.291577 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:30.303714 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:30.305167 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:30.315066 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d5e70413e5747b3bb06574770844127: Generated new TSK 0
I20260929 00:54:30.316104 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:30.368956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:30.690043 4036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:30.690368 4036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:30.690531 4036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:30.697750 4036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:30.697901 4036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:30.698076 4036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:30.707808 4036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:30.709105 4036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:30.710371 4036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:30.723362 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:30.723413 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:30.723387 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:30.724332 4036 server_base.cc:1034] running on GCE node
I20260929 00:54:30.725006 4036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:30.725885 4036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:30.727079 4036 hybrid_clock.cc:648] HybridClock initialized: now 1790643270727038 us; error 33 us; skew 500 ppm
I20260929 00:54:30.727388 4036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:30 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:30.730432 4036 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:30.731884 4036 webserver.cc:460] Webserver started at http://127.0.97.1:33531/ using document root <none> and password file <none>
I20260929 00:54:30.732784 4036 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:30.732925 4036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:30.733255 4036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:30.736074 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "b89e2ee3497b46069e8989f132a43598"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "39d79e61ac4a85d74bdcef3ffa81da78"
server_key_iv: "7db5f2908c54dce27ca1605194f041e1"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.736865 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b89e2ee3497b46069e8989f132a43598"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "39d79e61ac4a85d74bdcef3ffa81da78"
server_key_iv: "7db5f2908c54dce27ca1605194f041e1"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.741814 4036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 00:54:30.745065 4051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:30.746304 4036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:30.746475 4036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "b89e2ee3497b46069e8989f132a43598"
format_stamp: "Formatted at 2026-09-29 00:54:30 on dist-test-slave-pzsx"
server_key: "39d79e61ac4a85d74bdcef3ffa81da78"
server_key_iv: "7db5f2908c54dce27ca1605194f041e1"
server_key_version: "encryptionkey@0"
I20260929 00:54:30.746677 4036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:30.762120 4036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:30.766371 4036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:30.766656 4036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:30.767568 4036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:30.768855 4036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:30.768977 4036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:30.769129 4036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:30.769202 4036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:30.800987 4036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32935
I20260929 00:54:30.801033 4163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32935 every 8 connection(s)
I20260929 00:54:30.802239 4036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:30.808734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4036
I20260929 00:54:30.808948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:30.809544 388 external_mini_cluster.cc:1428] Setting key 13fdb44b8660affd61f6c515d0abf052
Sep 29 00:54:30 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:30.822419 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:30.804667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32883 (local address 127.0.97.62:42293)
0929 00:54:30.805098 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:30.805102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:30.805996 (+ 894us) server_negotiation.cc:407] Connection header received
0929 00:54:30.807722 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:30.807727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:30.807810 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:30.807923 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:30.809501 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.810396 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.812440 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.812887 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.812982 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:30.817064 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:30.817069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:30.817072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:30.817101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:30.819555 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:30.820254 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:30.820259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:30.820267 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:30.820369 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:30.820801 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:30.820806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:30.820809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:30.821057 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:30.821210 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:30.821843 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 00:54:30.822057 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":240,"threads_started":1}
I20260929 00:54:30.823371 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:30.805164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.1:32883)
0929 00:54:30.805840 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:30.805881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:30.807449 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:30.808150 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:30.808158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:30.808704 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:30.809338 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:30.809355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.810547 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.810551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:30.812298 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:30.812314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:30.813112 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:30.813122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:30.813450 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:30.814230 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:30.814262 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:30.816852 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:30.819710 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:30.819720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:30.819735 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:30.820121 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:30.820473 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:30.820479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:30.820483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:30.820686 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:30.821186 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:30.821193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:30.821509 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 00:54:30.821766 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 00:54:30.822269 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260929 00:54:30.825057 4164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260929 00:54:30.825357 4164 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:30.826054 4164 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:30.828089 3975 ts_manager.cc:194] Registered new tserver with Master: b89e2ee3497b46069e8989f132a43598 (127.0.97.1:32935)
I20260929 00:54:30.829674 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:30.853156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:31.170930 4170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:31.171209 4170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:31.171301 4170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:31.178431 4170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:31.178553 4170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:31.178670 4170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:31.188122 4170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:31.189427 4170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:31.190678 4170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:31.203979 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:31.204058 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:31.204034 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:31.204893 4170 server_base.cc:1034] running on GCE node
I20260929 00:54:31.205466 4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:31.206290 4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:31.207479 4170 hybrid_clock.cc:648] HybridClock initialized: now 1790643271207434 us; error 42 us; skew 500 ppm
I20260929 00:54:31.207793 4170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:31.210790 4170 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:31.212057 4170 webserver.cc:460] Webserver started at http://127.0.97.2:35999/ using document root <none> and password file <none>
I20260929 00:54:31.212756 4170 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:31.212854 4170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:31.213094 4170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:31.215768 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "123f3fa1128e4cd39cabbb6046913fa9"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "3fd92382ea7e16f51988d66dd1f63ecf"
server_key_iv: "671678a2ae8b0520a593b6ccb7524676"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.216481 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "123f3fa1128e4cd39cabbb6046913fa9"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "3fd92382ea7e16f51988d66dd1f63ecf"
server_key_iv: "671678a2ae8b0520a593b6ccb7524676"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.221374 4170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:31.224768 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.226004 4170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:31.226194 4170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "123f3fa1128e4cd39cabbb6046913fa9"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "3fd92382ea7e16f51988d66dd1f63ecf"
server_key_iv: "671678a2ae8b0520a593b6ccb7524676"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.226387 4170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:31.254102 4170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:31.258006 4170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:31.258296 4170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:31.259230 4170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:31.260486 4170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:31.260661 4170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.260812 4170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:31.260885 4170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.293128 4170 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38421
I20260929 00:54:31.293229 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38421 every 8 connection(s)
I20260929 00:54:31.294425 4170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:31.301931 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4170
I20260929 00:54:31.302145 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:31.302580 388 external_mini_cluster.cc:1428] Setting key 15f309a8c0543cdf33a2fc47fbdc14e5
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:31.315933 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.297024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34785 (local address 127.0.97.62:42293)
0929 00:54:31.297199 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:31.297203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:31.298324 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 00:54:31.300075 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:31.300081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:31.300167 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:31.300297 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:31.302078 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.302922 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.304886 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.305254 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.305345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.309409 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:31.309417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:31.309425 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:31.309468 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:31.312787 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.313710 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.313714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.313716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.313828 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.314296 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.314299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.314301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.314554 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:31.314663 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:31.315537 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 00:54:31.315703 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:54:31.317575 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.297455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.2:34785)
0929 00:54:31.298165 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:31.298208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:31.299794 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:31.300484 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:31.300492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:31.301117 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:31.301846 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:31.301881 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.303067 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.303075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:31.304757 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:31.304770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.305462 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.305471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:31.305833 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.306579 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:31.306613 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:31.309197 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:31.312918 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:31.312931 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:31.312951 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:31.313597 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:31.313937 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:31.313945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:31.313951 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:31.314196 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:31.314684 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:31.314695 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:31.315153 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 00:54:31.315445 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 00:54:31.316031 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260929 00:54:31.319116 4298 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260929 00:54:31.319428 4298 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:31.320398 4298 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
I20260929 00:54:31.322223 3975 ts_manager.cc:194] Registered new tserver with Master: 123f3fa1128e4cd39cabbb6046913fa9 (127.0.97.2:38421)
I20260929 00:54:31.323405 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34785
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:31.347970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:31.664768 4303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:31.665056 4303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:31.665247 4303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:31.672514 4303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:31.672644 4303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:31.672830 4303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:31.682422 4303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:31.683852 4303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:31.685148 4303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:31.697904 4308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:31.697881 4309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:31.697839 4311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:31.698647 4303 server_base.cc:1034] running on GCE node
I20260929 00:54:31.699285 4303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:31.700359 4303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:31.701545 4303 hybrid_clock.cc:648] HybridClock initialized: now 1790643271701495 us; error 80 us; skew 500 ppm
I20260929 00:54:31.701848 4303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:31.708958 4303 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:31.710291 4303 webserver.cc:460] Webserver started at http://127.0.97.3:34929/ using document root <none> and password file <none>
I20260929 00:54:31.710964 4303 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:31.711112 4303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:31.711410 4303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:31.714290 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "956420f8d19a48d39e0f28b41907538d"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "58666e1d317c539b07e151ad4975a4e5"
server_key_iv: "60777de528c9b0cd6f70a08b8b6c6550"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.715116 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "956420f8d19a48d39e0f28b41907538d"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "58666e1d317c539b07e151ad4975a4e5"
server_key_iv: "60777de528c9b0cd6f70a08b8b6c6550"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.719995 4303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:54:31.723491 4318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.724808 4303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:54:31.724932 4303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "956420f8d19a48d39e0f28b41907538d"
format_stamp: "Formatted at 2026-09-29 00:54:31 on dist-test-slave-pzsx"
server_key: "58666e1d317c539b07e151ad4975a4e5"
server_key_iv: "60777de528c9b0cd6f70a08b8b6c6550"
server_key_version: "encryptionkey@0"
I20260929 00:54:31.725087 4303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:31.749259 4303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:31.753600 4303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:31.753902 4303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:31.754846 4303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:31.756142 4303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:31.756212 4303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.756379 4303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:31.756430 4303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:31.787691 4303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45791
I20260929 00:54:31.787737 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45791 every 8 connection(s)
I20260929 00:54:31.789031 4303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:31.796172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4303
I20260929 00:54:31.796384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:31.796825 388 external_mini_cluster.cc:1428] Setting key 724c44371b5679b12dcb7b87635f8ecf
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:31.808897 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.791397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35435 (local address 127.0.97.62:42293)
0929 00:54:31.791570 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:31.791575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:31.792769 (+ 1194us) server_negotiation.cc:407] Connection header received
0929 00:54:31.794499 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:31.794504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:31.794592 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:31.794709 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:31.796327 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.797214 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.799144 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.799775 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.799867 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.803324 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:31.803330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:31.803335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:31.803379 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:31.806476 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.807099 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.807103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.807105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.807207 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.807540 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.807543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.807545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.807784 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:31.807885 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:31.808456 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 00:54:31.808672 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:54:31.809825 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.791890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42293 (local address 127.0.97.3:35435)
0929 00:54:31.792615 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:31.792658 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:31.794246 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:31.794933 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:31.794941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:31.795424 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:31.796106 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:31.796144 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.797383 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.797387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:31.799007 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:31.799018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.799966 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.799970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:31.800187 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.800875 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:31.800896 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:31.803121 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:31.806591 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:31.806597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:31.806609 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:31.806988 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:31.807294 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:31.807297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:31.807300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:31.807454 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:31.807885 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:31.807890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:31.808167 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 00:54:31.808374 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 00:54:31.808786 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":286,"threads_started":1}
I20260929 00:54:31.811182 4431 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42293
I20260929 00:54:31.811504 4431 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:31.812168 4431 heartbeater.cc:511] Master 127.0.97.62:42293 requested a full tablet report, sending...
I20260929 00:54:31.813491 3974 ts_manager.cc:194] Registered new tserver with Master: 956420f8d19a48d39e0f28b41907538d (127.0.97.3:45791)
I20260929 00:54:31.814337 3974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35435
I20260929 00:54:31.824040 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:31.832476 4164 heartbeater.cc:503] Master 127.0.97.62:42293 was elected leader, sending a full tablet report...
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:31.870280 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.856733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.97.1:32935)
0929 00:54:31.857362 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:31.857368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:31.857396 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:31.857454 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:31.857460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:31.857668 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:31.857793 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:31.859065 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.860020 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.861207 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.861733 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.861826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.865312 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:31.865322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:31.865336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:31.865369 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:31.867857 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.868459 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.868465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.868467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.868642 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.868950 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.868953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.868955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.869271 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:31.869477 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:31.869706 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:54:31.869917 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":343,"threads_started":1}
Sep 29 00:54:31 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:31.887452 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.874362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.0.97.62:42293)
0929 00:54:31.874569 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:31.874577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:31.874795 (+ 218us) server_negotiation.cc:407] Connection header received
0929 00:54:31.874973 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:31.874978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:31.875047 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:31.875145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:31.876298 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.877124 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.878321 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.878736 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.878870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.882510 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:31.882515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:31.882518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:31.882548 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:31.884981 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.885582 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.885589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.885594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.885723 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.886080 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.886086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.886090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.886397 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:31.886638 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:31.886945 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 00:54:31.887153 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 00:54:31.887993 3974 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072
I20260929 00:54:31.888973 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4036
I20260929 00:54:31.901458 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:31.889796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.97.1:32935)
0929 00:54:31.890015 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:31.890022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:31.890241 (+ 219us) server_negotiation.cc:407] Connection header received
0929 00:54:31.890432 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:31.890440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:31.890506 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:31.890618 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:31.891771 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.892445 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.893677 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:31.894101 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:31.894241 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:31.896539 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:31.896546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:31.896551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:31.896594 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:31.898891 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.899584 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.899590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.899593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.899713 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:31.900105 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:31.900110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:31.900113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:31.900347 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:31.900674 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:31.900925 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 00:54:31.901140 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 00:54:31.902009 4159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:31.983135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4036
I20260929 00:54:31.999346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4170
Sep 29 00:54:32 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:32.015081 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:32.000277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.0.97.2:38421)
0929 00:54:32.000931 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:32.000938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:32.000963 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:54:32.001021 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:32.001027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:32.001243 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:32.001415 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:32.002641 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:32.003668 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:32.004998 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:32.005547 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:32.005699 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:32.009759 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:32.009772 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:32.009788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:32.009822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:32.012795 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:32.013405 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:32.013412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:32.013414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:32.013555 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:32.013856 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:32.013859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:32.013861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:32.014187 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:32.014402 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:32.014607 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 00:54:32.014796 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":326,"threads_started":1}
I20260929 00:54:32.016363 4293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:32.103677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4170
I20260929 00:54:32.119450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4303
Sep 29 00:54:32 dist-test-slave-pzsx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:32.136281 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:32.120409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.97.3:45791)
0929 00:54:32.121107 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:32.121115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:32.121140 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:54:32.121198 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:32.121204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:32.121412 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:32.121585 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:32.122912 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:32.124043 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:32.125724 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:32.126269 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:32.126434 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:32.130883 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:32.130893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:32.130909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:32.130940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:32.133552 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:32.134192 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:32.134205 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:32.134207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:32.134324 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:32.134731 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:32.134736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:32.134739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:32.135083 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:32.135409 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:32.135668 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 00:54:32.135937 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":394,"threads_started":1}
I20260929 00:54:32.137578 4426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:32.223044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4303
I20260929 00:54:32.238880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 3943
I20260929 00:54:32.240051 4005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:32.341658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 3943
2026-09-29T00:54:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3805 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:32 dist-test-slave-pzsx krb5kdc[4450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:32 dist-test-slave-pzsx krb5kdc[4450](info): set up 2 sockets
Sep 29 00:54:32 dist-test-slave-pzsx krb5kdc[4450](info): commencing operation
krb5kdc: starting...
W20260929 00:54:33.413151 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:33 dist-test-slave-pzsx krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:33.604427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41895
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:33.916821 4463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:33.917116 4463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:33.917281 4463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:33.924476 4463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:33.924605 4463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:33.924654 4463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:33.924737 4463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:33.924808 4463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:33.934684 4463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:33.935971 4463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:33.937389 4463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:33.948813 4468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:33.948815 4471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:33.948844 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:33.949718 4463 server_base.cc:1034] running on GCE node
I20260929 00:54:33.950403 4463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:33.951748 4463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:33.952986 4463 hybrid_clock.cc:648] HybridClock initialized: now 1790643273952944 us; error 49 us; skew 500 ppm
I20260929 00:54:33.953342 4463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:33 dist-test-slave-pzsx krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:33.956220 4463 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:33.957479 4463 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:33.957607 4463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:33.957965 4463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:33.960968 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "0c8c73f42ae149cea592bfc6b4fc75ef"
format_stamp: "Formatted at 2026-09-29 00:54:33 on dist-test-slave-pzsx"
server_key: "68e4d5d1ff5e3297225167b4feb1a948"
server_key_iv: "43288631add0646da7c42d2200827f7f"
server_key_version: "encryptionkey@0"
I20260929 00:54:33.961755 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c8c73f42ae149cea592bfc6b4fc75ef"
format_stamp: "Formatted at 2026-09-29 00:54:33 on dist-test-slave-pzsx"
server_key: "68e4d5d1ff5e3297225167b4feb1a948"
server_key_iv: "43288631add0646da7c42d2200827f7f"
server_key_version: "encryptionkey@0"
I20260929 00:54:33.966692 4463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:33.970041 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:33.971308 4463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:33.971495 4463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0c8c73f42ae149cea592bfc6b4fc75ef"
format_stamp: "Formatted at 2026-09-29 00:54:33 on dist-test-slave-pzsx"
server_key: "68e4d5d1ff5e3297225167b4feb1a948"
server_key_iv: "43288631add0646da7c42d2200827f7f"
server_key_version: "encryptionkey@0"
I20260929 00:54:33.971707 4463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:33.984700 4463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:33.988476 4463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:33.988790 4463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:34.006182 4463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41895
I20260929 00:54:34.006207 4528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41895 every 8 connection(s)
I20260929 00:54:34.007036 4463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:34.010566 4529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:34.012717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4463
I20260929 00:54:34.012930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:34.013700 388 external_mini_cluster.cc:1428] Setting key 42cefffbd57418bd087b4d9ed49b8362
I20260929 00:54:34.022763 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: Bootstrap starting.
I20260929 00:54:34.025902 4529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:34.026875 4529 log.cc:824] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:54:34 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:34.029873 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: No bootstrap required, opened a new log
I20260929 00:54:34.037613 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:34.015667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.62:41895)
0929 00:54:34.016431 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:34.016442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:34.016556 (+ 114us) server_negotiation.cc:407] Connection header received
0929 00:54:34.018182 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:34.018202 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:34.018565 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:34.018884 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:34.020680 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.021925 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.023753 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.024460 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.024744 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:34.028797 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:34.028806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:34.028824 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:34.028865 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:34.032694 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:34.033536 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:34.033543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:34.033552 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:34.033746 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:34.034166 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:34.034176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:34.034180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:34.034828 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:34.035162 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:34.035602 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 00:54:34.036221 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":370,"threads_started":1}
I20260929 00:54:34.040374 4529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } }
I20260929 00:54:34.040828 4529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:34.040948 4529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c73f42ae149cea592bfc6b4fc75ef, State: Initialized, Role: FOLLOWER
I20260929 00:54:34.041471 4529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } }
I20260929 00:54:34.041702 4529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:34.041843 4529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:34.042011 4529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:34.044342 4529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } }
I20260929 00:54:34.044850 4529 leader_election.cc:302] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8c73f42ae149cea592bfc6b4fc75ef; no voters:
I20260929 00:54:34.045410 4529 leader_election.cc:288] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:34.045537 4534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:34.046258 4534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [term 1 LEADER]: Becoming Leader. State: Replica: 0c8c73f42ae149cea592bfc6b4fc75ef, State: Running, Role: LEADER
I20260929 00:54:34.046716 4534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } }
I20260929 00:54:34.047490 4529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:34.049371 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } }
I20260929 00:54:34.049770 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:34.050297 4536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8c73f42ae149cea592bfc6b4fc75ef. Latest consensus state: current_term: 1 leader_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8c73f42ae149cea592bfc6b4fc75ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } }
I20260929 00:54:34.050637 4536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:34.057812 4542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:34.061858 4542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:34.069787 4542 catalog_manager.cc:1347] Generated new cluster ID: 9973e3fa7e1d4653a441279e882cfd73
I20260929 00:54:34.069905 4542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:34.087826 4542 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:34.089216 4542 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:34.100100 4542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: Generated new TSK 0
I20260929 00:54:34.100915 4542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:34.156414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:34.480211 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:34.480481 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:34.480597 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:34.487811 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:34.487929 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:34.488044 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:34.497772 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:34.498965 4555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:34.500156 4555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:34.512733 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:34.512759 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:34.513245 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:34.513468 4555 server_base.cc:1034] running on GCE node
I20260929 00:54:34.514026 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:34.514892 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:34.516114 4555 hybrid_clock.cc:648] HybridClock initialized: now 1790643274516069 us; error 39 us; skew 500 ppm
I20260929 00:54:34.516424 4555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:34 dist-test-slave-pzsx krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:34.519336 4555 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:34.520488 4555 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:34.520682 4555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:34.521008 4555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:34.523803 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "07b081c515b5414abc35bfde68b199e2"
format_stamp: "Formatted at 2026-09-29 00:54:34 on dist-test-slave-pzsx"
server_key: "5988762c94355fcaf37e9cdac0017cbe"
server_key_iv: "d33c09ff03f6dffaef83b73b382da800"
server_key_version: "encryptionkey@0"
I20260929 00:54:34.524688 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "07b081c515b5414abc35bfde68b199e2"
format_stamp: "Formatted at 2026-09-29 00:54:34 on dist-test-slave-pzsx"
server_key: "5988762c94355fcaf37e9cdac0017cbe"
server_key_iv: "d33c09ff03f6dffaef83b73b382da800"
server_key_version: "encryptionkey@0"
I20260929 00:54:34.529537 4555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:54:34.532879 4569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:34.534174 4555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:54:34.534343 4555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "07b081c515b5414abc35bfde68b199e2"
format_stamp: "Formatted at 2026-09-29 00:54:34 on dist-test-slave-pzsx"
server_key: "5988762c94355fcaf37e9cdac0017cbe"
server_key_iv: "d33c09ff03f6dffaef83b73b382da800"
server_key_version: "encryptionkey@0"
I20260929 00:54:34.534551 4555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:34.564682 4555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:34.568476 4555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:34.568804 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:34.569868 4555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:34.571357 4555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:34.571439 4555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:34.571513 4555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:34.571566 4555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:34.604012 4555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38095
I20260929 00:54:34.604058 4681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38095 every 8 connection(s)
I20260929 00:54:34.604959 4555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:34.615161 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4555
I20260929 00:54:34.615377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:34.616212 388 external_mini_cluster.cc:1428] Setting key 73a25c06be1f75e0d954b6f0ea2b5694
Sep 29 00:54:34 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:34.624969 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:34.607301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43181 (local address 127.0.97.62:41895)
0929 00:54:34.607759 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:34.607765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:34.608776 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 00:54:34.610474 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:34.610478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:34.610542 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:34.610641 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:34.612017 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.612783 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.614239 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.614601 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.614694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:34.618953 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:34.618957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:34.618961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:34.618988 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:34.621487 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:34.622414 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:34.622417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:34.622420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:34.622529 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:34.623003 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:34.623006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:34.623008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:34.623262 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:34.623366 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:34.624269 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 00:54:34.624535 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":217,"threads_started":1}
I20260929 00:54:34.625650 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:34.607896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.97.1:43181)
0929 00:54:34.608616 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:34.608661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:34.610239 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:34.610834 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:34.610841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:34.611336 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:34.611880 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:34.611896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.612911 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.612914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:34.614117 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:34.614127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:34.614836 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:34.614845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:34.615137 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:34.615883 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:34.615922 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:34.618733 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:34.621630 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:34.621649 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:34.621669 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:34.622304 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:34.622641 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:34.622649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:34.622655 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:34.622901 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:34.623403 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:34.623414 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:34.623855 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 00:54:34.624101 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 00:54:34.624474 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":275,"threads_started":1}
I20260929 00:54:34.627197 4682 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41895
I20260929 00:54:34.627496 4682 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:34.628139 4682 heartbeater.cc:511] Master 127.0.97.62:41895 requested a full tablet report, sending...
I20260929 00:54:34.630240 4494 ts_manager.cc:194] Registered new tserver with Master: 07b081c515b5414abc35bfde68b199e2 (127.0.97.1:38095)
I20260929 00:54:34.631747 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43181
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:34.659894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:34.971688 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:34.971956 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:34.972123 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:34.979342 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:34.979471 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:34.979648 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:34.989172 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:34.990435 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:34.991726 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:35.004537 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:35.004485 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:35.004568 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:35.005218 4688 server_base.cc:1034] running on GCE node
I20260929 00:54:35.005897 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:35.006942 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:35.008167 4688 hybrid_clock.cc:648] HybridClock initialized: now 1790643275008114 us; error 47 us; skew 500 ppm
I20260929 00:54:35.008491 4688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:35 dist-test-slave-pzsx krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:35.011759 4688 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:35.013025 4688 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:35.013163 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:35.013486 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:35.016320 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "2b69a38e77c24bf9a30b515d624bf51a"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "5f9f60f4254bc5ccd3d77353eb025483"
server_key_iv: "9f3a86f2585beae751a52b1e98eb28f7"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.017253 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2b69a38e77c24bf9a30b515d624bf51a"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "5f9f60f4254bc5ccd3d77353eb025483"
server_key_iv: "9f3a86f2585beae751a52b1e98eb28f7"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.021989 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:54:35.025208 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.026449 4688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:35.026611 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "2b69a38e77c24bf9a30b515d624bf51a"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "5f9f60f4254bc5ccd3d77353eb025483"
server_key_iv: "9f3a86f2585beae751a52b1e98eb28f7"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.026834 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:35.046583 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:35.050765 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:35.051060 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:35.052060 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:35.053629 4688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:35.053751 4688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.053884 4688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:35.053972 4688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.086246 4688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32831
I20260929 00:54:35.086297 4814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32831 every 8 connection(s)
I20260929 00:54:35.087116 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:35.088289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4688
I20260929 00:54:35.088516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:35.089123 388 external_mini_cluster.cc:1428] Setting key 75b54ade0f61efe6f9fd5979c1287ea9
Sep 29 00:54:35 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:35.107051 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.089762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44449 (local address 127.0.97.62:41895)
0929 00:54:35.089941 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.089948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.091199 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 00:54:35.093036 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.093041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.093130 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.093244 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:35.094805 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.095540 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.097239 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.097660 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.097797 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.101318 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:35.101323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:35.101326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:35.101357 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:35.103920 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.104709 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.104718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.104720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.104834 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.105256 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.105261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.105264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.105532 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:35.105682 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.106504 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.106672 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:54:35.108083 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.090298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.97.2:44449)
0929 00:54:35.091047 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.091092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.092779 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.093461 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.093470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.093971 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:35.094536 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.094552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.095698 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.095705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.097088 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.097100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.097898 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.097905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.098125 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.098832 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:35.098853 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:35.101089 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:35.104079 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:35.104088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:35.104105 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:35.104583 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:35.104953 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:35.104958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:35.104962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:35.105136 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:35.105660 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:35.105668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:35.106101 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.106376 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.106872 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:54:35.109721 4815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41895
I20260929 00:54:35.110059 4815 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:35.110845 4815 heartbeater.cc:511] Master 127.0.97.62:41895 requested a full tablet report, sending...
I20260929 00:54:35.112435 4494 ts_manager.cc:194] Registered new tserver with Master: 2b69a38e77c24bf9a30b515d624bf51a (127.0.97.2:32831)
I20260929 00:54:35.113241 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:35.132561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:35.446636 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:35.446905 4820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:35.447077 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:35.454706 4820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:35.454864 4820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:35.455051 4820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:35.465009 4820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:35.466355 4820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:35.467674 4820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:35.480348 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:35.480412 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:35.480448 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:35.481094 4820 server_base.cc:1034] running on GCE node
I20260929 00:54:35.481693 4820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:35.482649 4820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:35.483870 4820 hybrid_clock.cc:648] HybridClock initialized: now 1790643275483824 us; error 45 us; skew 500 ppm
I20260929 00:54:35.484205 4820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:35 dist-test-slave-pzsx krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:35.487408 4820 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:35.488569 4820 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:35.488687 4820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:35.489033 4820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:35.491818 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "a48c0d8cb4b6895c6c2576304a34df97"
server_key_iv: "87b7a06f5feff67b91842d3115c48de0"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.492676 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "a48c0d8cb4b6895c6c2576304a34df97"
server_key_iv: "87b7a06f5feff67b91842d3115c48de0"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.497779 4820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:54:35.500993 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.502352 4820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:35.502519 4820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817"
format_stamp: "Formatted at 2026-09-29 00:54:35 on dist-test-slave-pzsx"
server_key: "a48c0d8cb4b6895c6c2576304a34df97"
server_key_iv: "87b7a06f5feff67b91842d3115c48de0"
server_key_version: "encryptionkey@0"
I20260929 00:54:35.502758 4820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:35.515465 4820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:35.519688 4820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:35.519968 4820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:35.520949 4820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:35.522200 4820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:35.522318 4820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.522449 4820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:35.522535 4820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:35.555467 4820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44495
I20260929 00:54:35.555509 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44495 every 8 connection(s)
I20260929 00:54:35.556336 4820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:35.560388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4820
I20260929 00:54:35.560638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:35.561089 388 external_mini_cluster.cc:1428] Setting key 8ea627a69e9ca376460f5c1a601ef5bd
Sep 29 00:54:35 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:35.575305 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.558810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46481 (local address 127.0.97.62:41895)
0929 00:54:35.558982 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.558987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.560117 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 00:54:35.562029 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.562034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.562122 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.562247 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:35.563639 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.564431 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.565958 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.566349 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.566481 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.569751 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:35.569757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:35.569762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:35.569802 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:35.572279 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.573021 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.573029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.573034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.573162 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.573568 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.573573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.573577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.573828 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:35.573978 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.574689 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.574897 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:35.576218 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.559315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.97.3:46481)
0929 00:54:35.559968 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.560005 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.561773 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.562464 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.562472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.562960 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:35.563484 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.563500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.564599 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.564606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.565809 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.565820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.566563 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.566573 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.566809 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.567499 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:35.567522 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:35.569562 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:35.572418 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:35.572427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:35.572442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:35.572904 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:35.573280 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:35.573285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:35.573288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:35.573455 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:35.573953 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:35.573960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:35.574287 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.574542 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.575057 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":274,"threads_started":1}
I20260929 00:54:35.577481 4947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41895
I20260929 00:54:35.577742 4947 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:35.578310 4947 heartbeater.cc:511] Master 127.0.97.62:41895 requested a full tablet report, sending...
I20260929 00:54:35.579591 4494 ts_manager.cc:194] Registered new tserver with Master: 6fb9cc78d03c45b3a683dc2c9e3ea817 (127.0.97.3:44495)
I20260929 00:54:35.580358 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46481
I20260929 00:54:35.588109 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:35.604055 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.592414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.97.62:41895)
0929 00:54:35.592586 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.592591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.593116 (+ 525us) server_negotiation.cc:407] Connection header received
0929 00:54:35.593259 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.593262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.593318 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.593396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:35.594759 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.595442 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.596656 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.597072 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.597170 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.599236 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:35.599243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:35.599248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:35.599283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:35.601793 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.602336 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.602339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.602341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.602439 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.602778 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.602782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.602784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.603054 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:35.603235 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.603530 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.603737 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:54:35.608376 4494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:35.615607 4494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:35.634439 4682 heartbeater.cc:503] Master 127.0.97.62:41895 was elected leader, sending a full tablet report...
I20260929 00:54:35.638307 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.627502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.97.3:44495)
0929 00:54:35.629189 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.629200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.629835 (+ 635us) server_negotiation.cc:407] Connection header received
0929 00:54:35.629947 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.629957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.630225 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.630411 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:35.631652 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.633274 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.635977 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.637094 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.637231 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.637452 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.637745 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.637959 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1471,"thread_start_us":1211,"threads_started":1}
I20260929 00:54:35.639168 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.626823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44495 (local address 127.0.0.1:38092)
0929 00:54:35.629448 (+ 2625us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.629489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.629652 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.630717 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.630728 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.630767 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:35.631452 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.631470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.633403 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.633410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.635852 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.635863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.637299 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.637305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.637496 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.637546 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.637689 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.638793 (+ 1104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2409,"thread_start_us":2249,"threads_started":1}
I20260929 00:54:35.639743 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.626828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38095 (local address 127.0.0.1:40062)
0929 00:54:35.631512 (+ 4684us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.631526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.631664 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.632362 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.632370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.632399 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:35.632819 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.632827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.634654 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.634662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.636830 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.636843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.638878 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.638883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.639067 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.639122 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.639203 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.639323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4569,"mutex_wait_us":26,"spinlock_wait_cycles":18304,"thread_start_us":4385,"threads_started":1}
I20260929 00:54:35.640348 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.627168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.97.1:38095)
0929 00:54:35.629730 (+ 2562us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.629741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.631793 (+ 2052us) server_negotiation.cc:407] Connection header received
0929 00:54:35.631890 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.631898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.632119 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.632289 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:35.632986 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.634538 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.636982 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.638405 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.638522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.638746 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.639369 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.640057 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2354,"thread_start_us":3198,"threads_started":1}
I20260929 00:54:35.640967 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.629828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32831 (local address 127.0.0.1:55210)
0929 00:54:35.632398 (+ 2570us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.632414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.632564 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.633108 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.633113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.633142 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:35.633502 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.633517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.636007 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.636012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.638192 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.638203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.640213 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.640217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.640353 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.640382 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.640470 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.640624 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2451,"spinlock_wait_cycles":5248,"thread_start_us":2338,"threads_started":1}
I20260929 00:54:35.641659 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.630228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.97.2:32831)
0929 00:54:35.631973 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.631984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.632590 (+ 606us) server_negotiation.cc:407] Connection header received
0929 00:54:35.632697 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.632704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.632889 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.633051 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:35.633693 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.635847 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.638363 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.639830 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.639954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.640109 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.641231 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.641385 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1587,"thread_start_us":1500,"threads_started":1}
I20260929 00:54:35.643231 4882 tablet_service.cc:1467] Processing CreateTablet for tablet 5cea2470598e473696d59ce30b12be82 (DEFAULT_TABLE table=test-table [id=59991a8c83e54e3488313f9f53c447f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:35.643584 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 5cea2470598e473696d59ce30b12be82 (DEFAULT_TABLE table=test-table [id=59991a8c83e54e3488313f9f53c447f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:35.644721 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cea2470598e473696d59ce30b12be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:35.644744 4882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cea2470598e473696d59ce30b12be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:35.644889 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 5cea2470598e473696d59ce30b12be82 (DEFAULT_TABLE table=test-table [id=59991a8c83e54e3488313f9f53c447f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:35.646098 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cea2470598e473696d59ce30b12be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:35.656899 4965 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Bootstrap starting.
I20260929 00:54:35.658757 4966 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Bootstrap starting.
I20260929 00:54:35.660218 4965 tablet_bootstrap.cc:654] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:35.661194 4967 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Bootstrap starting.
I20260929 00:54:35.661528 4965 log.cc:824] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:35.662456 4966 tablet_bootstrap.cc:654] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:35.663609 4966 log.cc:824] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:35.664162 4967 tablet_bootstrap.cc:654] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:35.665127 4965 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: No bootstrap required, opened a new log
I20260929 00:54:35.665405 4967 log.cc:824] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:35.665441 4965 ts_tablet_manager.cc:1397] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 00:54:35.666512 4966 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: No bootstrap required, opened a new log
I20260929 00:54:35.666795 4966 ts_tablet_manager.cc:1397] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 00:54:35.668279 4967 tablet_bootstrap.cc:492] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: No bootstrap required, opened a new log
I20260929 00:54:35.668563 4967 ts_tablet_manager.cc:1397] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 00:54:35.672799 4965 raft_consensus.cc:348] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.673154 4965 raft_consensus.cc:374] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:35.673252 4965 raft_consensus.cc:729] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b081c515b5414abc35bfde68b199e2, State: Initialized, Role: FOLLOWER
I20260929 00:54:35.673866 4965 consensus_queue.cc:260] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.677436 4966 raft_consensus.cc:348] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.677744 4965 ts_tablet_manager.cc:1428] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260929 00:54:35.677917 4966 raft_consensus.cc:374] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:35.678057 4966 raft_consensus.cc:729] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb9cc78d03c45b3a683dc2c9e3ea817, State: Initialized, Role: FOLLOWER
I20260929 00:54:35.678714 4966 consensus_queue.cc:260] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.678748 4967 raft_consensus.cc:348] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.679138 4967 raft_consensus.cc:374] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:35.679327 4967 raft_consensus.cc:729] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b69a38e77c24bf9a30b515d624bf51a, State: Initialized, Role: FOLLOWER
I20260929 00:54:35.679824 4947 heartbeater.cc:503] Master 127.0.97.62:41895 was elected leader, sending a full tablet report...
I20260929 00:54:35.680032 4967 consensus_queue.cc:260] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.680414 4966 ts_tablet_manager.cc:1428] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 00:54:35.681360 4815 heartbeater.cc:503] Master 127.0.97.62:41895 was elected leader, sending a full tablet report...
I20260929 00:54:35.681944 4967 ts_tablet_manager.cc:1428] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260929 00:54:35.796308 4972 raft_consensus.cc:482] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:35.796670 4972 raft_consensus.cc:504] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.798403 4972 leader_election.cc:288] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07b081c515b5414abc35bfde68b199e2 (127.0.97.1:38095), 2b69a38e77c24bf9a30b515d624bf51a (127.0.97.2:32831)
I20260929 00:54:35.804440 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.798795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44455 (local address 127.0.97.2:32831)
0929 00:54:35.798983 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.798990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.799014 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:54:35.799234 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.799238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.799294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.799395 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:35.799852 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.800826 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.802606 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.803676 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.803769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.803830 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.804181 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.804260 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:35.804615 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.798702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32831 (local address 127.0.97.3:44455)
0929 00:54:35.798922 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.798943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.799067 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.799412 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.799417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.799442 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:35.799690 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.799718 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.800949 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.800955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.802465 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.802475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.803822 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.803827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.804023 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.804058 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.804157 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.804289 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260929 00:54:35.805490 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.798702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38095 (local address 127.0.97.3:59443)
0929 00:54:35.799498 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:35.799517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:35.799650 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:35.799984 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:35.799988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:35.800011 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:35.800357 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.800367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.801659 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.801662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.803305 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:35.803318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.804824 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.804830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:35.805008 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.805042 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:54:35.805131 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 00:54:35.805254 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":178,"thread_start_us":302,"threads_started":1}
I20260929 00:54:35.805537 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.798917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59443 (local address 127.0.97.1:38095)
0929 00:54:35.799117 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.799125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.799615 (+ 490us) server_negotiation.cc:407] Connection header received
0929 00:54:35.799786 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.799795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.799874 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.799940 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:35.800568 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.801518 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.803454 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.804467 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.804615 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.804729 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.805169 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.805314 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:54:35.805750 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cea2470598e473696d59ce30b12be82" candidate_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b69a38e77c24bf9a30b515d624bf51a" is_pre_election: true
I20260929 00:54:35.806253 4770 raft_consensus.cc:2454] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb9cc78d03c45b3a683dc2c9e3ea817 in term 0.
I20260929 00:54:35.806514 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cea2470598e473696d59ce30b12be82" candidate_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b081c515b5414abc35bfde68b199e2" is_pre_election: true
I20260929 00:54:35.806955 4838 leader_election.cc:302] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b69a38e77c24bf9a30b515d624bf51a, 6fb9cc78d03c45b3a683dc2c9e3ea817; no voters:
I20260929 00:54:35.807024 4637 raft_consensus.cc:2454] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb9cc78d03c45b3a683dc2c9e3ea817 in term 0.
I20260929 00:54:35.807358 4972 raft_consensus.cc:2781] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:35.807492 4972 raft_consensus.cc:482] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:35.807641 4972 raft_consensus.cc:3037] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 0 FOLLOWER]: Advancing to term 1
W20260929 00:54:35.807862 4948 tablet.cc:2369] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:35.809993 4972 raft_consensus.cc:504] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.810612 4972 leader_election.cc:288] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [CANDIDATE]: Term 1 election: Requested vote from peers 07b081c515b5414abc35bfde68b199e2 (127.0.97.1:38095), 2b69a38e77c24bf9a30b515d624bf51a (127.0.97.2:32831)
I20260929 00:54:35.810951 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cea2470598e473696d59ce30b12be82" candidate_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b081c515b5414abc35bfde68b199e2"
I20260929 00:54:35.811033 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cea2470598e473696d59ce30b12be82" candidate_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b69a38e77c24bf9a30b515d624bf51a"
I20260929 00:54:35.811128 4637 raft_consensus.cc:3037] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:35.811278 4770 raft_consensus.cc:3037] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:35.813323 4637 raft_consensus.cc:2454] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb9cc78d03c45b3a683dc2c9e3ea817 in term 1.
I20260929 00:54:35.813493 4770 raft_consensus.cc:2454] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb9cc78d03c45b3a683dc2c9e3ea817 in term 1.
I20260929 00:54:35.813746 4836 leader_election.cc:302] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b081c515b5414abc35bfde68b199e2, 6fb9cc78d03c45b3a683dc2c9e3ea817; no voters:
I20260929 00:54:35.814051 4972 raft_consensus.cc:2781] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:35.814456 4972 raft_consensus.cc:686] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb9cc78d03c45b3a683dc2c9e3ea817, State: Running, Role: LEADER
I20260929 00:54:35.814930 4972 consensus_queue.cc:237] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } }
I20260929 00:54:35.819499 4493 catalog_manager.cc:5495] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fb9cc78d03c45b3a683dc2c9e3ea817 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fb9cc78d03c45b3a683dc2c9e3ea817" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 } health_report { overall_health: UNKNOWN } } }
W20260929 00:54:35.839025 4816 tablet.cc:2369] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:35.857215 4683 tablet.cc:2369] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:35.874053 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.868470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.97.3:44495)
0929 00:54:35.868713 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.868718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.868736 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:54:35.868949 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.868953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.869027 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.869192 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:35.869563 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.870265 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.872007 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.872392 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.872539 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.873027 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:35.873500 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:35.873671 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.873722 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.873796 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 00:54:35.882099 4770 raft_consensus.cc:1261] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 1 FOLLOWER]: Refusing update from remote peer 6fb9cc78d03c45b3a683dc2c9e3ea817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:35.882109 4637 raft_consensus.cc:1261] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 1 FOLLOWER]: Refusing update from remote peer 6fb9cc78d03c45b3a683dc2c9e3ea817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:35.883119 4975 consensus_queue.cc:1035] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b69a38e77c24bf9a30b515d624bf51a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:35.883443 4972 consensus_queue.cc:1035] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b081c515b5414abc35bfde68b199e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:35.894050 4978 mvcc.cc:205] Tried to move back new op lower bound from 7334474858002661376 to 7334474857742053376. Current Snapshot: MvccSnapshot[applied={T|T < 7334474858002661376}]
I20260929 00:54:35.906782 4494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:35.907102 4494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:35.910257 4494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c8c73f42ae149cea592bfc6b4fc75ef: Sending DeleteTablet for 3 replicas of tablet 5cea2470598e473696d59ce30b12be82
I20260929 00:54:35.911352 4617 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cea2470598e473696d59ce30b12be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40062
I20260929 00:54:35.911360 4882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cea2470598e473696d59ce30b12be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38092
I20260929 00:54:35.912269 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cea2470598e473696d59ce30b12be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55210
I20260929 00:54:35.912537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4555
I20260929 00:54:35.926689 4989 tablet_replica.cc:330] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: stopping tablet replica
I20260929 00:54:35.927254 4989 raft_consensus.cc:2229] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:35.927898 4989 raft_consensus.cc:2258] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:35.932024 4987 tablet_replica.cc:330] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: stopping tablet replica
I20260929 00:54:35.932399 4987 raft_consensus.cc:2229] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:35.932861 4988 tablet_replica.cc:330] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: stopping tablet replica
I20260929 00:54:35.933183 4988 raft_consensus.cc:2229] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:35.933583 4988 raft_consensus.cc:2258] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:35.934091 4989 ts_tablet_manager.cc:1905] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:35.934384 4987 raft_consensus.cc:2258] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:35.935931 4988 ts_tablet_manager.cc:1905] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:35.939528 4987 ts_tablet_manager.cc:1905] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:54:35 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:35.944283 4989 ts_tablet_manager.cc:1918] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:35.944442 4989 log.cc:1196] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/5cea2470598e473696d59ce30b12be82
I20260929 00:54:35.944855 4987 ts_tablet_manager.cc:1918] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:35.945000 4989 ts_tablet_manager.cc:1939] T 5cea2470598e473696d59ce30b12be82 P 2b69a38e77c24bf9a30b515d624bf51a: Deleting consensus metadata
I20260929 00:54:35.945091 4987 log.cc:1196] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/5cea2470598e473696d59ce30b12be82
I20260929 00:54:35.945761 4987 ts_tablet_manager.cc:1939] T 5cea2470598e473696d59ce30b12be82 P 6fb9cc78d03c45b3a683dc2c9e3ea817: Deleting consensus metadata
I20260929 00:54:35.948004 4481 catalog_manager.cc:4841] TS 6fb9cc78d03c45b3a683dc2c9e3ea817 (127.0.97.3:44495): tablet 5cea2470598e473696d59ce30b12be82 (table test-table [id=59991a8c83e54e3488313f9f53c447f6]) successfully deleted
I20260929 00:54:35.949690 4988 ts_tablet_manager.cc:1918] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:35.949903 4988 log.cc:1196] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/5cea2470598e473696d59ce30b12be82
I20260929 00:54:35.950465 4988 ts_tablet_manager.cc:1939] T 5cea2470598e473696d59ce30b12be82 P 07b081c515b5414abc35bfde68b199e2: Deleting consensus metadata
I20260929 00:54:35.950758 4481 catalog_manager.cc:4841] TS 2b69a38e77c24bf9a30b515d624bf51a (127.0.97.2:32831): tablet 5cea2470598e473696d59ce30b12be82 (table test-table [id=59991a8c83e54e3488313f9f53c447f6]) successfully deleted
I20260929 00:54:35.951889 4479 catalog_manager.cc:4841] TS 07b081c515b5414abc35bfde68b199e2 (127.0.97.1:38095): tablet 5cea2470598e473696d59ce30b12be82 (table test-table [id=59991a8c83e54e3488313f9f53c447f6]) successfully deleted
I20260929 00:54:35.954222 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:35.914356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.97.1:38095)
0929 00:54:35.914531 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:35.914537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:35.923812 (+ 9275us) server_negotiation.cc:407] Connection header received
0929 00:54:35.924286 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:35.924292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:35.924385 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:35.924461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:35.928128 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.929383 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.931764 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:35.932407 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:35.932532 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:35.946288 (+ 13756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:35.946306 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:35.946323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:35.946381 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:35.950977 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.951848 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.951858 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.951861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.952017 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:35.952447 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:35.952453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:35.952456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:35.952924 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:35.953291 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:35.953612 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 00:54:35.953841 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:54:35.954797 4677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:36.059841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4555
I20260929 00:54:36.083308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4688
Sep 29 00:54:36 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:36.102676 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:36.084103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.97.2:32831)
0929 00:54:36.084272 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:36.084276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:36.084293 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:54:36.084489 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:36.084493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:36.084598 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:36.084688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:36.086252 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:36.087465 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:36.089574 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:36.090233 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:36.090357 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:36.095439 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:36.095453 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:36.095473 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:36.095526 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:36.099492 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:36.100283 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:36.100293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:36.100297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:36.100440 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:36.100992 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:36.100997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:36.101001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:36.101462 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:36.101738 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:36.102052 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 00:54:36.102279 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:54:36.103421 4810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:36.191893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4688
I20260929 00:54:36.215267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4820
Sep 29 00:54:36 dist-test-slave-pzsx krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:36.233987 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:36.216123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.0.97.3:44495)
0929 00:54:36.216298 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:36.216302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:36.216364 (+ 62us) server_negotiation.cc:407] Connection header received
0929 00:54:36.216583 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:36.216587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:36.216657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:36.216741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:36.218456 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:36.219812 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:36.221709 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:36.222146 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:36.222237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:36.227335 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:36.227349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:36.227368 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:36.227407 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:36.230673 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:36.231405 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:36.231419 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:36.231421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:36.231572 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:36.231965 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:36.231972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:36.231978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:36.232328 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:36.232704 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:36.233534 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 00:54:36.233699 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:54:36.234823 4942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:36.357379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4820
I20260929 00:54:36.376276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4463
I20260929 00:54:36.377562 4524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:36.486495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4463
2026-09-29T00:54:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4144 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T00:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:36Z Disabled control of system clock
I20260929 00:54:36.551894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44365
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:54:36.875206 4998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:36.875505 4998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:36.875586 4998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:36.883044 4998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:36.883147 4998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:36.883185 4998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:36.883265 4998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:36.883327 4998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:36.892571 4998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:36.893853 4998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:36.895998 4998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:36.908242 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:36.908210 5006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:36.908267 5003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:36.909142 4998 server_base.cc:1034] running on GCE node
I20260929 00:54:36.909946 4998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:36.911309 4998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:36.912586 4998 hybrid_clock.cc:648] HybridClock initialized: now 1790643276912519 us; error 44 us; skew 500 ppm
I20260929 00:54:36.912964 4998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:36.921691 4998 webserver.cc:460] Webserver started at http://127.0.97.62:43921/ using document root <none> and password file <none>
I20260929 00:54:36.922438 4998 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:36.922585 4998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:36.922926 4998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:36.925896 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "8f765cc993ed4634add5769d1e71e616"
format_stamp: "Formatted at 2026-09-29 00:54:36 on dist-test-slave-pzsx"
server_key: "43ca61a808eb06adbe800e87c41306ea"
server_key_iv: "b98f486ddb5a983c80aed9580c8f4e86"
server_key_version: "encryptionkey@0"
I20260929 00:54:36.926708 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f765cc993ed4634add5769d1e71e616"
format_stamp: "Formatted at 2026-09-29 00:54:36 on dist-test-slave-pzsx"
server_key: "43ca61a808eb06adbe800e87c41306ea"
server_key_iv: "b98f486ddb5a983c80aed9580c8f4e86"
server_key_version: "encryptionkey@0"
I20260929 00:54:36.931887 4998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 00:54:36.935467 5012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:36.936874 4998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:54:36.937055 4998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "8f765cc993ed4634add5769d1e71e616"
format_stamp: "Formatted at 2026-09-29 00:54:36 on dist-test-slave-pzsx"
server_key: "43ca61a808eb06adbe800e87c41306ea"
server_key_iv: "b98f486ddb5a983c80aed9580c8f4e86"
server_key_version: "encryptionkey@0"
I20260929 00:54:36.937278 4998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:36.942800 4998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:36.943536 4998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:36.943845 4998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:36.961588 4998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44365
I20260929 00:54:36.961623 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44365 every 8 connection(s)
I20260929 00:54:36.962914 4998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:36.966497 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:36.970278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 4998
I20260929 00:54:36.970463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:36.971241 388 external_mini_cluster.cc:1428] Setting key 69e04b8222c12c8794aa24adee392cc0
I20260929 00:54:36.979205 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:36.973275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.0.97.62:44365)
0929 00:54:36.974225 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:36.974241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:36.974280 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:54:36.975760 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:36.975783 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:36.976196 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:36.976614 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:36.977679 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:36.977684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:36.977696 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:36.977711 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:36.977765 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:36.977779 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:36.978005 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:36.978137 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 00:54:36.978376 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":340,"threads_started":1}
I20260929 00:54:36.979295 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: Bootstrap starting.
I20260929 00:54:36.982383 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:36.983340 5064 log.cc:824] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:36.985824 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: No bootstrap required, opened a new log
I20260929 00:54:36.992791 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } }
I20260929 00:54:36.993166 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:36.993274 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f765cc993ed4634add5769d1e71e616, State: Initialized, Role: FOLLOWER
I20260929 00:54:36.993805 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } }
I20260929 00:54:36.994051 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:36.994194 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:36.994361 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:36.996409 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } }
I20260929 00:54:36.996933 5064 leader_election.cc:302] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f765cc993ed4634add5769d1e71e616; no voters:
I20260929 00:54:36.997478 5064 leader_election.cc:288] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:36.997609 5069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:36.998307 5069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [term 1 LEADER]: Becoming Leader. State: Replica: 8f765cc993ed4634add5769d1e71e616, State: Running, Role: LEADER
I20260929 00:54:36.998780 5069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } }
I20260929 00:54:36.999514 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:37.001528 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f765cc993ed4634add5769d1e71e616. Latest consensus state: current_term: 1 leader_uuid: "8f765cc993ed4634add5769d1e71e616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } } }
I20260929 00:54:37.001870 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:37.002174 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f765cc993ed4634add5769d1e71e616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f765cc993ed4634add5769d1e71e616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44365 } } }
I20260929 00:54:37.002729 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:37.003163 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:37.008271 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:37.016983 5077 catalog_manager.cc:1347] Generated new cluster ID: 9c82e91bcce0478ab673528e5161b265
I20260929 00:54:37.017115 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:37.026739 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:37.027251 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:37.037406 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: Generated new TSK 0
I20260929 00:54:37.038447 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:54:37.053117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--builtin_ntp_servers=127.0.97.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 00:54:37.378397 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:37.378788 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:37.378985 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:37.386137 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:37.386279 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:37.386487 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:37.395375 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:37.396701 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:37.398487 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:37.411880 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:37.411760 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:37.411888 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:37.412686 5088 server_base.cc:1034] running on GCE node
I20260929 00:54:37.414363 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:37.415521 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:37.416726 5088 hybrid_clock.cc:648] HybridClock initialized: now 1790643277416657 us; error 67 us; skew 500 ppm
I20260929 00:54:37.417053 5088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:37.419217 5088 webserver.cc:460] Webserver started at http://127.0.97.1:45289/ using document root <none> and password file <none>
I20260929 00:54:37.419914 5088 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:37.420051 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:37.420382 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:37.423331 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "2dd92224765948dca5301d3101938899"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "e744e2259c95f46853d4f86d6b534ca3"
server_key_iv: "43c5e8525ba9c1a3a5b3e1939101a8c1"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.424126 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2dd92224765948dca5301d3101938899"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "e744e2259c95f46853d4f86d6b534ca3"
server_key_iv: "43c5e8525ba9c1a3a5b3e1939101a8c1"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.429150 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:54:37.432473 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.433813 5088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:37.433986 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "2dd92224765948dca5301d3101938899"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "e744e2259c95f46853d4f86d6b534ca3"
server_key_iv: "43c5e8525ba9c1a3a5b3e1939101a8c1"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.434227 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:37.439968 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:37.440891 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:37.441253 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:37.442248 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:37.443626 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:37.443754 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.443893 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:37.443984 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.476682 5088 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37005
I20260929 00:54:37.476728 5214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37005 every 8 connection(s)
I20260929 00:54:37.478014 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:37.482367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5088
I20260929 00:54:37.482553 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:37.483222 388 external_mini_cluster.cc:1428] Setting key cd6ec80fb6bfde4279fed24741796689
I20260929 00:54:37.486133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--builtin_ntp_servers=127.0.97.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 00:54:37.487082 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:37.480915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38711 (local address 127.0.97.62:44365)
0929 00:54:37.481363 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:37.481367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:37.482682 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 00:54:37.484611 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:37.484617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:37.484680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:37.484812 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:37.486142 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:37.486148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:37.486157 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:37.486173 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:37.486212 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:37.486224 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:37.486354 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:37.486632 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 00:54:37.486701 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":234,"threads_started":1}
I20260929 00:54:37.487854 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:37.481541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44365 (local address 127.0.97.1:38711)
0929 00:54:37.482476 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:37.482527 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:37.484309 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:37.485059 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:37.485067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:37.485161 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:37.485848 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:37.485851 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:37.485887 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:37.485973 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:37.486406 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:37.486420 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:37.486425 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 00:54:37.486637 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 00:54:37.486838 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
I20260929 00:54:37.489766 5215 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44365
I20260929 00:54:37.490123 5215 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:37.490743 5215 heartbeater.cc:511] Master 127.0.97.62:44365 requested a full tablet report, sending...
I20260929 00:54:37.493206 5029 ts_manager.cc:194] Registered new tserver with Master: 2dd92224765948dca5301d3101938899 (127.0.97.1:37005)
W20260929 00:54:37.802521 5219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:37.802863 5219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:37.803035 5219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:37.810386 5219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:37.810529 5219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:37.810722 5219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:37.819707 5219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:37.821027 5219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:37.822791 5219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:37.835453 5224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:37.835484 5225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:37.835513 5227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:37.836798 5219 server_base.cc:1034] running on GCE node
I20260929 00:54:37.837381 5219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:37.838465 5219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:37.839686 5219 hybrid_clock.cc:648] HybridClock initialized: now 1790643277839637 us; error 45 us; skew 500 ppm
I20260929 00:54:37.840006 5219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:37.842366 5219 webserver.cc:460] Webserver started at http://127.0.97.2:35285/ using document root <none> and password file <none>
I20260929 00:54:37.843056 5219 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:37.843187 5219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:37.843494 5219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:37.846341 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "d1fd06cfd4254b28b25f77da9666d71e"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "5600447154f83f357dcf91f8cf460b1b"
server_key_iv: "7ec743dc283d5b40d47486456f6032b6"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.847132 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d1fd06cfd4254b28b25f77da9666d71e"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "5600447154f83f357dcf91f8cf460b1b"
server_key_iv: "7ec743dc283d5b40d47486456f6032b6"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.852118 5219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:54:37.855568 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.857023 5219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:37.857200 5219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "d1fd06cfd4254b28b25f77da9666d71e"
format_stamp: "Formatted at 2026-09-29 00:54:37 on dist-test-slave-pzsx"
server_key: "5600447154f83f357dcf91f8cf460b1b"
server_key_iv: "7ec743dc283d5b40d47486456f6032b6"
server_key_version: "encryptionkey@0"
I20260929 00:54:37.857429 5219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:37.862910 5219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:37.863672 5219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:37.863931 5219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:37.864820 5219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:37.866127 5219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:37.866243 5219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.866367 5219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:37.866453 5219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:37.898963 5219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42797
I20260929 00:54:37.899020 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42797 every 8 connection(s)
I20260929 00:54:37.900300 5219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:37.905311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5219
I20260929 00:54:37.905505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:37.906000 388 external_mini_cluster.cc:1428] Setting key 7c2a6e5b7ed2151f57e5bbd2e56c2131
I20260929 00:54:37.908648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--builtin_ntp_servers=127.0.97.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 00:54:37.908731 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:37.902947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55923 (local address 127.0.97.62:44365)
0929 00:54:37.903135 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:37.903140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:37.904557 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 00:54:37.906460 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:37.906466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:37.906535 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:37.906657 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:37.907955 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:37.907960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:37.907965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:37.907983 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:37.908023 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:37.908034 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:37.908155 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:37.908353 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 00:54:37.908417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:54:37.909895 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:37.903493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44365 (local address 127.0.97.2:55923)
0929 00:54:37.904376 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:37.904417 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:37.906187 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:37.906884 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:37.906892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:37.906987 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:37.907673 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:37.907675 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:37.907712 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:37.907794 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:37.908129 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:37.908139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:37.908146 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:54:37.908426 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 00:54:37.908705 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":262,"threads_started":1}
I20260929 00:54:37.911392 5346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44365
I20260929 00:54:37.911715 5346 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:37.912297 5346 heartbeater.cc:511] Master 127.0.97.62:44365 requested a full tablet report, sending...
I20260929 00:54:37.913831 5029 ts_manager.cc:194] Registered new tserver with Master: d1fd06cfd4254b28b25f77da9666d71e (127.0.97.2:42797)
W20260929 00:54:38.239141 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:38.239419 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:38.239508 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:38.247010 5349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:38.247130 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:38.247242 5349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:38.256428 5349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:38.257838 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:38.259563 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:38.272583 5357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:38.272629 5355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:38.272562 5354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:38.278142 5349 server_base.cc:1034] running on GCE node
I20260929 00:54:38.278828 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:38.279865 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:38.281112 5349 hybrid_clock.cc:648] HybridClock initialized: now 1790643278281072 us; error 52 us; skew 500 ppm
I20260929 00:54:38.281464 5349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:38.283756 5349 webserver.cc:460] Webserver started at http://127.0.97.3:33225/ using document root <none> and password file <none>
I20260929 00:54:38.284545 5349 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:38.284691 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:38.285013 5349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:38.287892 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "cdd97022350444fd97e6e5ae131c4477"
format_stamp: "Formatted at 2026-09-29 00:54:38 on dist-test-slave-pzsx"
server_key: "01df69ed9f6924e91082ec49df1f3848"
server_key_iv: "6214190a91a60007305ce3aaba3c617c"
server_key_version: "encryptionkey@0"
I20260929 00:54:38.288841 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cdd97022350444fd97e6e5ae131c4477"
format_stamp: "Formatted at 2026-09-29 00:54:38 on dist-test-slave-pzsx"
server_key: "01df69ed9f6924e91082ec49df1f3848"
server_key_iv: "6214190a91a60007305ce3aaba3c617c"
server_key_version: "encryptionkey@0"
I20260929 00:54:38.294126 5349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:38.297659 5363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:38.299000 5349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:38.299181 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "cdd97022350444fd97e6e5ae131c4477"
format_stamp: "Formatted at 2026-09-29 00:54:38 on dist-test-slave-pzsx"
server_key: "01df69ed9f6924e91082ec49df1f3848"
server_key_iv: "6214190a91a60007305ce3aaba3c617c"
server_key_version: "encryptionkey@0"
I20260929 00:54:38.299449 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:38.305027 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:38.305784 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:38.306038 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:38.306910 5349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:38.308238 5349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:38.308367 5349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:38.308492 5349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:38.308621 5349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 00:54:38.342094 5349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37875
I20260929 00:54:38.342147 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37875 every 8 connection(s)
I20260929 00:54:38.343446 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:38.349697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5349
I20260929 00:54:38.349891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:38.350361 388 external_mini_cluster.cc:1428] Setting key 2bf543c7b5430ec33aa8c663f5351262
I20260929 00:54:38.351811 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.345911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43157 (local address 127.0.97.62:44365)
0929 00:54:38.346126 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.346132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.347696 (+ 1564us) server_negotiation.cc:407] Connection header received
0929 00:54:38.349405 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.349413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.349487 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.349605 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.351059 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.351065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.351070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.351100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.351137 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.351149 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.351293 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.351492 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.351556 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:38.352909 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.346629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44365 (local address 127.0.97.3:43157)
0929 00:54:38.347531 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.347571 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.349137 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.349886 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.349902 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.350042 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.350754 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.350756 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.350793 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.350873 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.351267 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.351279 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.351287 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.351586 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.351860 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260929 00:54:38.354166 5476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44365
I20260929 00:54:38.354415 5476 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:38.355059 5476 heartbeater.cc:511] Master 127.0.97.62:44365 requested a full tablet report, sending...
I20260929 00:54:38.356410 5029 ts_manager.cc:194] Registered new tserver with Master: cdd97022350444fd97e6e5ae131c4477 (127.0.97.3:37875)
I20260929 00:54:38.365784 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:38.372625 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.97.62:44365)
0929 00:54:38.370395 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.370400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.370941 (+ 541us) server_negotiation.cc:407] Connection header received
0929 00:54:38.371034 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.371037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.371088 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.371174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.372039 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.372043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.372046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.372061 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.372090 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.372102 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.372204 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.372266 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.372306 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:54:38.376559 5029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:38.383203 5029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:38.412163 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.395853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.97.2:42797)
0929 00:54:38.406156 (+ 10303us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.406170 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.408394 (+ 2224us) server_negotiation.cc:407] Connection header received
0929 00:54:38.408740 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.408751 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.409048 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.409174 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.411212 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.411223 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.411239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.411270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.411327 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.411345 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.411472 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.411718 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.411818 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10040,"thread_start_us":10317,"threads_started":1}
I20260929 00:54:38.412326 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.395466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42797 (local address 127.0.0.1:35008)
0929 00:54:38.408266 (+ 12800us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.408286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.408491 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.410321 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.410328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.410352 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.411001 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.411004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.411016 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.411051 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.411525 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.411531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.411537 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.411734 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.411821 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12646,"thread_start_us":11142,"threads_started":1}
I20260929 00:54:38.412803 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.395173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37875 (local address 127.0.0.1:35370)
0929 00:54:38.407139 (+ 11966us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.407195 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.407417 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.409904 (+ 2487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.409912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.410021 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.410874 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.410879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.410911 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.411039 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.411514 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.411524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.411531 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.411993 (+ 462us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.412044 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11618,"thread_start_us":13411,"threads_started":1}
I20260929 00:54:38.413550 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.398256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42692)
0929 00:54:38.408825 (+ 10569us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.408842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.408928 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.410327 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.410332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.410357 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.411774 (+ 1417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.411777 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.411790 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.411825 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.412818 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.412823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.412827 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.413053 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.413101 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10463,"thread_start_us":9695,"threads_started":1}
I20260929 00:54:38.413540 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.395468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.0.97.3:37875)
0929 00:54:38.408632 (+ 13164us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.408640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.408676 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:38.408745 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.408753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.409048 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.409204 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.411248 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.411259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.411274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.411304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.411370 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.411393 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.411592 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.413039 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.413145 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12940,"thread_start_us":12745,"threads_started":1}
I20260929 00:54:38.414215 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.398946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.0.97.1:37005)
0929 00:54:38.407791 (+ 8845us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.407802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.409073 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 00:54:38.409156 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.409163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.409341 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.409562 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.412491 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.412521 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.412532 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.412559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.412613 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.412634 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.412761 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.413899 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.413990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8652,"thread_start_us":8531,"threads_started":1}
I20260929 00:54:38.416021 5281 tablet_service.cc:1467] Processing CreateTablet for tablet b4f680d4a4fb4570b6198b36e022569b (DEFAULT_TABLE table=test-table [id=d6a11b30e0314d8899b1445b1c6c6838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:38.417562 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f680d4a4fb4570b6198b36e022569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:38.418212 5411 tablet_service.cc:1467] Processing CreateTablet for tablet b4f680d4a4fb4570b6198b36e022569b (DEFAULT_TABLE table=test-table [id=d6a11b30e0314d8899b1445b1c6c6838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:38.419144 5150 tablet_service.cc:1467] Processing CreateTablet for tablet b4f680d4a4fb4570b6198b36e022569b (DEFAULT_TABLE table=test-table [id=d6a11b30e0314d8899b1445b1c6c6838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:38.419697 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f680d4a4fb4570b6198b36e022569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:38.420370 5150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f680d4a4fb4570b6198b36e022569b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:38.432930 5494 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Bootstrap starting.
I20260929 00:54:38.433797 5495 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Bootstrap starting.
I20260929 00:54:38.435104 5496 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Bootstrap starting.
I20260929 00:54:38.437145 5494 tablet_bootstrap.cc:654] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:38.437289 5495 tablet_bootstrap.cc:654] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:38.438354 5496 tablet_bootstrap.cc:654] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:38.438467 5494 log.cc:824] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:38.438467 5495 log.cc:824] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:38.439535 5496 log.cc:824] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:38.442559 5494 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: No bootstrap required, opened a new log
I20260929 00:54:38.442971 5494 ts_tablet_manager.cc:1397] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 00:54:38.447026 5495 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: No bootstrap required, opened a new log
I20260929 00:54:38.447301 5495 ts_tablet_manager.cc:1397] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 00:54:38.448549 5496 tablet_bootstrap.cc:492] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: No bootstrap required, opened a new log
I20260929 00:54:38.448794 5496 ts_tablet_manager.cc:1397] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260929 00:54:38.454867 5495 raft_consensus.cc:348] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.455250 5495 raft_consensus.cc:374] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:38.454981 5494 raft_consensus.cc:348] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.455339 5495 raft_consensus.cc:729] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fd06cfd4254b28b25f77da9666d71e, State: Initialized, Role: FOLLOWER
I20260929 00:54:38.455379 5494 raft_consensus.cc:374] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:38.455473 5494 raft_consensus.cc:729] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd97022350444fd97e6e5ae131c4477, State: Initialized, Role: FOLLOWER
I20260929 00:54:38.456045 5495 consensus_queue.cc:260] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.456149 5494 consensus_queue.cc:260] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.457151 5346 heartbeater.cc:503] Master 127.0.97.62:44365 was elected leader, sending a full tablet report...
I20260929 00:54:38.457824 5495 ts_tablet_manager.cc:1428] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260929 00:54:38.458906 5476 heartbeater.cc:503] Master 127.0.97.62:44365 was elected leader, sending a full tablet report...
I20260929 00:54:38.459362 5494 ts_tablet_manager.cc:1428] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260929 00:54:38.459731 5496 raft_consensus.cc:348] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.460170 5496 raft_consensus.cc:374] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:38.460271 5496 raft_consensus.cc:729] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd92224765948dca5301d3101938899, State: Initialized, Role: FOLLOWER
I20260929 00:54:38.460942 5496 consensus_queue.cc:260] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.462210 5215 heartbeater.cc:503] Master 127.0.97.62:44365 was elected leader, sending a full tablet report...
I20260929 00:54:38.462872 5496 ts_tablet_manager.cc:1428] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 00:54:38.480486 5216 tablet.cc:2369] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:38.595335 5477 tablet.cc:2369] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:38.652432 5347 tablet.cc:2369] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:38.678972 5501 raft_consensus.cc:482] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:38.679302 5501 raft_consensus.cc:504] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.681049 5501 leader_election.cc:288] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1fd06cfd4254b28b25f77da9666d71e (127.0.97.2:42797), 2dd92224765948dca5301d3101938899 (127.0.97.1:37005)
I20260929 00:54:38.689719 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.681337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36795 (local address 127.0.97.2:42797)
0929 00:54:38.681514 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.681521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.681757 (+ 236us) server_negotiation.cc:407] Connection header received
0929 00:54:38.681925 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.681933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.681996 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.682113 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.689034 (+ 6921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.689042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.689049 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.689071 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.689120 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.689133 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.689225 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.689368 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.689432 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:54:38.689745 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.681337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42797 (local address 127.0.97.3:36795)
0929 00:54:38.681611 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.681640 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.681781 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.682177 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.682184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.682210 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.688814 (+ 6604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.688817 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.688832 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.688871 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.689234 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.689240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.689248 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.689353 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.689434 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 00:54:38.689780 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.681441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57769 (local address 127.0.97.1:37005)
0929 00:54:38.681642 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.681650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.682244 (+ 594us) server_negotiation.cc:407] Connection header received
0929 00:54:38.682353 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.682356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.682401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.682510 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.689110 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.689114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.689117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.689133 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.689200 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.689220 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.689338 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.689469 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.689526 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:54:38.690001 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.681437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.97.3:57769)
0929 00:54:38.682135 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:38.682150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:38.682245 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:38.682531 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:38.682536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:38.682558 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:38.688863 (+ 6305us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:54:38.688866 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:38.688879 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:54:38.688914 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:38.689343 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:38.689348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:38.689352 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 00:54:38.689470 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 00:54:38.689513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":393,"threads_started":1}
I20260929 00:54:38.690860 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f680d4a4fb4570b6198b36e022569b" candidate_uuid: "cdd97022350444fd97e6e5ae131c4477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd92224765948dca5301d3101938899" is_pre_election: true
I20260929 00:54:38.690856 5301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f680d4a4fb4570b6198b36e022569b" candidate_uuid: "cdd97022350444fd97e6e5ae131c4477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fd06cfd4254b28b25f77da9666d71e" is_pre_election: true
I20260929 00:54:38.691371 5170 raft_consensus.cc:2454] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd97022350444fd97e6e5ae131c4477 in term 0.
I20260929 00:54:38.691377 5301 raft_consensus.cc:2454] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd97022350444fd97e6e5ae131c4477 in term 0.
I20260929 00:54:38.691982 5364 leader_election.cc:302] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd92224765948dca5301d3101938899, cdd97022350444fd97e6e5ae131c4477; no voters:
I20260929 00:54:38.692376 5501 raft_consensus.cc:2781] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:38.692534 5501 raft_consensus.cc:482] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:38.692672 5501 raft_consensus.cc:3037] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:38.694773 5501 raft_consensus.cc:504] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.695415 5501 leader_election.cc:288] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [CANDIDATE]: Term 1 election: Requested vote from peers d1fd06cfd4254b28b25f77da9666d71e (127.0.97.2:42797), 2dd92224765948dca5301d3101938899 (127.0.97.1:37005)
I20260929 00:54:38.695784 5301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f680d4a4fb4570b6198b36e022569b" candidate_uuid: "cdd97022350444fd97e6e5ae131c4477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fd06cfd4254b28b25f77da9666d71e"
I20260929 00:54:38.695909 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f680d4a4fb4570b6198b36e022569b" candidate_uuid: "cdd97022350444fd97e6e5ae131c4477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd92224765948dca5301d3101938899"
I20260929 00:54:38.695989 5301 raft_consensus.cc:3037] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:38.696131 5170 raft_consensus.cc:3037] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:38.698210 5301 raft_consensus.cc:2454] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd97022350444fd97e6e5ae131c4477 in term 1.
I20260929 00:54:38.698275 5170 raft_consensus.cc:2454] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd97022350444fd97e6e5ae131c4477 in term 1.
I20260929 00:54:38.698629 5367 leader_election.cc:302] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdd97022350444fd97e6e5ae131c4477, d1fd06cfd4254b28b25f77da9666d71e; no voters:
I20260929 00:54:38.698939 5501 raft_consensus.cc:2781] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:38.699419 5501 raft_consensus.cc:686] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 1 LEADER]: Becoming Leader. State: Replica: cdd97022350444fd97e6e5ae131c4477, State: Running, Role: LEADER
I20260929 00:54:38.699928 5501 consensus_queue.cc:237] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } }
I20260929 00:54:38.704195 5028 catalog_manager.cc:5495] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd97022350444fd97e6e5ae131c4477 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cdd97022350444fd97e6e5ae131c4477" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd97022350444fd97e6e5ae131c4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:38.787554 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.785570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.97.3:37875)
0929 00:54:38.785794 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.785798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.785834 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:54:38.786028 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.786031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.786106 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.786193 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.787008 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.787012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.787015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.787028 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.787057 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.787068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.787168 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.787221 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.787262 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 00:54:38.795331 5170 raft_consensus.cc:1261] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 1 FOLLOWER]: Refusing update from remote peer cdd97022350444fd97e6e5ae131c4477: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:38.795331 5301 raft_consensus.cc:1261] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 1 FOLLOWER]: Refusing update from remote peer cdd97022350444fd97e6e5ae131c4477: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:38.796243 5504 consensus_queue.cc:1035] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1fd06cfd4254b28b25f77da9666d71e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:38.796613 5501 consensus_queue.cc:1035] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd92224765948dca5301d3101938899" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:38.807572 5507 mvcc.cc:205] Tried to move back new op lower bound from 7334474869936599040 to 7334474869558726656. Current Snapshot: MvccSnapshot[applied={T|T < 7334474869936599040}]
I20260929 00:54:38.808960 5509 mvcc.cc:205] Tried to move back new op lower bound from 7334474869936599040 to 7334474869558726656. Current Snapshot: MvccSnapshot[applied={T|T < 7334474869936599040}]
I20260929 00:54:38.811120 5508 mvcc.cc:205] Tried to move back new op lower bound from 7334474869936599040 to 7334474869558726656. Current Snapshot: MvccSnapshot[applied={T|T < 7334474869936599040}]
I20260929 00:54:38.819957 5027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:38.820216 5027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:38.823649 5027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f765cc993ed4634add5769d1e71e616: Sending DeleteTablet for 3 replicas of tablet b4f680d4a4fb4570b6198b36e022569b
I20260929 00:54:38.824771 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f680d4a4fb4570b6198b36e022569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:38 UTC) from {username='slave'} at 127.0.0.1:35370
I20260929 00:54:38.824869 5281 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f680d4a4fb4570b6198b36e022569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:38 UTC) from {username='slave'} at 127.0.0.1:35008
I20260929 00:54:38.825536 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f680d4a4fb4570b6198b36e022569b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:38 UTC) from {username='slave'} at 127.0.0.1:42692
I20260929 00:54:38.825789 5517 tablet_replica.cc:330] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: stopping tablet replica
I20260929 00:54:38.826227 5517 raft_consensus.cc:2229] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:38.828490 5517 raft_consensus.cc:2258] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:38.832011 5517 ts_tablet_manager.cc:1905] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:38.838706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5088
I20260929 00:54:38.841634 5517 ts_tablet_manager.cc:1918] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:38.841961 5517 log.cc:1196] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/b4f680d4a4fb4570b6198b36e022569b
I20260929 00:54:38.842811 5517 ts_tablet_manager.cc:1939] T b4f680d4a4fb4570b6198b36e022569b P cdd97022350444fd97e6e5ae131c4477: Deleting consensus metadata
I20260929 00:54:38.845109 5014 catalog_manager.cc:4841] TS cdd97022350444fd97e6e5ae131c4477 (127.0.97.3:37875): tablet b4f680d4a4fb4570b6198b36e022569b (table test-table [id=d6a11b30e0314d8899b1445b1c6c6838]) successfully deleted
I20260929 00:54:38.864897 5519 tablet_replica.cc:330] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899: stopping tablet replica
I20260929 00:54:38.866839 5519 raft_consensus.cc:2229] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:38.867540 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:38.841224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.0.97.1:37005)
0929 00:54:38.841405 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:38.841411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:38.863963 (+ 22552us) server_negotiation.cc:407] Connection header received
0929 00:54:38.864209 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:38.864214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:38.864339 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:38.864464 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:38.866737 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:38.866743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:38.866747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:38.866769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:38.866807 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:38.866829 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:38.866962 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:38.867191 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 00:54:38.867253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:54:38.868618 5210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:38.867519 5519 raft_consensus.cc:2258] T b4f680d4a4fb4570b6198b36e022569b P 2dd92224765948dca5301d3101938899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:38.877833 5518 tablet_replica.cc:330] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: stopping tablet replica
I20260929 00:54:38.878369 5518 raft_consensus.cc:2229] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:38.879047 5518 raft_consensus.cc:2258] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:38.881647 5518 ts_tablet_manager.cc:1905] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:38.889971 5518 ts_tablet_manager.cc:1918] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:38.890228 5518 log.cc:1196] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/b4f680d4a4fb4570b6198b36e022569b
I20260929 00:54:38.890884 5518 ts_tablet_manager.cc:1939] T b4f680d4a4fb4570b6198b36e022569b P d1fd06cfd4254b28b25f77da9666d71e: Deleting consensus metadata
I20260929 00:54:38.893133 5016 catalog_manager.cc:4841] TS d1fd06cfd4254b28b25f77da9666d71e (127.0.97.2:42797): tablet b4f680d4a4fb4570b6198b36e022569b (table test-table [id=d6a11b30e0314d8899b1445b1c6c6838]) successfully deleted
I20260929 00:54:38.994902 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5088
W20260929 00:54:39.012125 5013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37005 (error 108)
I20260929 00:54:39.012372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5219
I20260929 00:54:39.014441 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.013664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42724)
0929 00:54:39.014093 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.014219 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":221,"threads_started":1}
W20260929 00:54:39.014717 5013 catalog_manager.cc:4584] TS 2dd92224765948dca5301d3101938899 (127.0.97.1:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4f680d4a4fb4570b6198b36e022569b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
I20260929 00:54:39.015687 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.013203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.0.97.2:42797)
0929 00:54:39.013379 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:39.013387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:39.013452 (+ 65us) server_negotiation.cc:407] Connection header received
0929 00:54:39.013668 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:39.013676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:39.013791 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:39.013903 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:39.015042 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:39.015054 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:39.015058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:39.015072 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:39.015101 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:39.015112 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:39.015204 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:39.015349 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 00:54:39.015389 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:54:39.016358 5341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:39.068684 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.068088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42736)
0929 00:54:39.068354 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.068479 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 00:54:39.069924 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.069525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42746)
0929 00:54:39.069682 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.069764 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:54:39.104167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5219
I20260929 00:54:39.120715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5349
I20260929 00:54:39.123624 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.121654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.0.97.3:37875)
0929 00:54:39.121821 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:39.121825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:39.121845 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:54:39.121916 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:39.121919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:39.121969 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:39.122049 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:39.122948 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:39.122952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:39.122955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:39.122969 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:39.123000 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:39.123011 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:39.123122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:39.123251 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 00:54:39.123294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:54:39.124359 5471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:39.130234 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.129845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42752)
0929 00:54:39.130011 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.130096 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:54:39.131253 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.130935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42762)
0929 00:54:39.131053 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.131122 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 00:54:39.211931 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.211470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42776)
0929 00:54:39.211699 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.211791 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 00:54:39.212982 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.212606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37005 (local address 127.0.0.1:42784)
0929 00:54:39.212773 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.212855 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 00:54:39.241312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5349
I20260929 00:54:39.259925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 4998
I20260929 00:54:39.261054 5059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:39.364653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 4998
2026-09-29T00:54:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2877 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 00:54:39.405326 388 mini_oidc.cc:152] Starting JWKS server
I20260929 00:54:39.407861 388 webserver.cc:460] Webserver started at https://127.0.0.1:42653/ using document root <none> and password file <none>
I20260929 00:54:39.408177 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:54:39.408861 388 webserver.cc:460] Webserver started at http://127.0.0.1:33483/ using document root <none> and password file <none>
2026-09-29T00:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:39Z Disabled control of system clock
I20260929 00:54:39.435349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43957
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43957
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/testchainca.cert with env {}
W20260929 00:54:39.751153 5532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:39.751493 5532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:39.751631 5532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:39.758637 5532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:39.758741 5532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:39.758778 5532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:39.758826 5532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:39.758869 5532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:54:39.758924 5532 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:54:39.758961 5532 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42653/jwks.json
I20260929 00:54:39.768730 5532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:39.769979 5532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:39.771179 5532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:39.782859 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:39.782917 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:39.782882 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:39.783912 5532 server_base.cc:1034] running on GCE node
I20260929 00:54:39.784744 5532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:39.786095 5532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:39.787286 5532 hybrid_clock.cc:648] HybridClock initialized: now 1790643279787229 us; error 48 us; skew 500 ppm
I20260929 00:54:39.787638 5532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:39.789980 5532 webserver.cc:460] Webserver started at http://127.0.97.62:36199/ using document root <none> and password file <none>
I20260929 00:54:39.790657 5532 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:39.790776 5532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:39.791088 5532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:39.793969 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "4ea82b72ba504e098fe3bc616f9004ea"
format_stamp: "Formatted at 2026-09-29 00:54:39 on dist-test-slave-pzsx"
server_key: "6f2d31944fb012289638e2f687984b14"
server_key_iv: "61d1eec3efa0cb1734b4b98d97194e0e"
server_key_version: "encryptionkey@0"
I20260929 00:54:39.794754 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "4ea82b72ba504e098fe3bc616f9004ea"
format_stamp: "Formatted at 2026-09-29 00:54:39 on dist-test-slave-pzsx"
server_key: "6f2d31944fb012289638e2f687984b14"
server_key_iv: "61d1eec3efa0cb1734b4b98d97194e0e"
server_key_version: "encryptionkey@0"
I20260929 00:54:39.799631 5532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 00:54:39.803123 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:39.804301 5532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:39.804533 5532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "4ea82b72ba504e098fe3bc616f9004ea"
format_stamp: "Formatted at 2026-09-29 00:54:39 on dist-test-slave-pzsx"
server_key: "6f2d31944fb012289638e2f687984b14"
server_key_iv: "61d1eec3efa0cb1734b4b98d97194e0e"
server_key_version: "encryptionkey@0"
I20260929 00:54:39.804767 5532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:39.825873 5532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:39.826689 5532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:39.826982 5532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:39.845391 5532 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43957
I20260929 00:54:39.845397 5597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43957 every 8 connection(s)
I20260929 00:54:39.846665 5532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:39.850411 5598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:39.853478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5532
I20260929 00:54:39.853703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:39.854305 388 external_mini_cluster.cc:1428] Setting key 45071bbe659a3802bc12c8dcadb2613e
I20260929 00:54:39.861927 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea: Bootstrap starting.
I20260929 00:54:39.864856 5598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:39.865826 5598 log.cc:824] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:39.867441 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.856103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.0.97.62:43957)
0929 00:54:39.857081 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:39.857092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:39.857132 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:54:39.858677 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:39.858698 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:39.859154 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:39.859565 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:39.860132 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.861413 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.862807 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.863468 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.863706 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:39.864948 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:39.864981 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:39.865001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:39.865547 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:39.865634 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:39.865658 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:39.865787 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:39.865990 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 00:54:39.866301 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":381,"threads_started":1}
I20260929 00:54:39.868824 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea: No bootstrap required, opened a new log
I20260929 00:54:39.871220 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:54:39.876267 5598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260929 00:54:39.876658 5598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:39.876771 5598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea82b72ba504e098fe3bc616f9004ea, State: Initialized, Role: FOLLOWER
I20260929 00:54:39.877349 5598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260929 00:54:39.877627 5598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:39.877735 5598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:39.877843 5598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:39.879729 5598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260929 00:54:39.880110 5598 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea82b72ba504e098fe3bc616f9004ea; no voters:
I20260929 00:54:39.880638 5598 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:39.880769 5603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:39.881434 5603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [term 1 LEADER]: Becoming Leader. State: Replica: 4ea82b72ba504e098fe3bc616f9004ea, State: Running, Role: LEADER
I20260929 00:54:39.881958 5603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260929 00:54:39.882637 5598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:39.884958 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea82b72ba504e098fe3bc616f9004ea. Latest consensus state: current_term: 1 leader_uuid: "4ea82b72ba504e098fe3bc616f9004ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } } }
I20260929 00:54:39.885311 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:39.885071 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea82b72ba504e098fe3bc616f9004ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea82b72ba504e098fe3bc616f9004ea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } } }
I20260929 00:54:39.885465 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:39.886809 5609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:39.891918 5609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:39.901033 5609 catalog_manager.cc:1347] Generated new cluster ID: 28612f0563a04d699dca9ba6107f7c3b
I20260929 00:54:39.901139 5609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:39.923812 5609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:39.925208 5609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:39.933795 5609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea82b72ba504e098fe3bc616f9004ea: Generated new TSK 0
I20260929 00:54:39.934684 5609 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 00:54:39.977948 5628 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 00:54:39.978119 5628 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:54:39.979293 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:39.957895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.97.62:43957)
0929 00:54:39.958109 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:39.958121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:39.958506 (+ 385us) server_negotiation.cc:407] Connection header received
0929 00:54:39.958751 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:39.958757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:39.958859 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:39.959012 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:54:39.959478 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.960707 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.962089 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.962795 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.962919 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:39.963411 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:54:39.978631 (+ 15220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:54:39.978995 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 00:54:39.979418 5601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 00:54:39.980767 5627 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:39.957732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43957 (local address 127.0.0.1:52524)
0929 00:54:39.958388 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:39.958405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:39.958547 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:39.959043 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:39.959047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:39.959056 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:54:39.959336 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:39.959343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.960833 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.960837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:39.961951 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:39.961962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:39.962915 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:39.962954 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:39.963084 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:39.963139 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:54:39.978925 (+ 15786us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 00:54:39.979173 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":348,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:43957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 00:54:39.982961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5532
I20260929 00:54:39.984205 5593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:40.079887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5532
2026-09-29T00:54:40Z chronyd exiting
I20260929 00:54:40.130488 388 test_util.cc:175] -----------------------------------------------
I20260929 00:54:40.130667 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643237839665-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 00:54:40.135342 388 mini_oidc.cc:152] Starting JWKS server
I20260929 00:54:40.137225 388 webserver.cc:460] Webserver started at https://127.0.0.1:45783/ using document root <none> and password file <none>
I20260929 00:54:40.137451 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:54:40.138046 388 webserver.cc:460] Webserver started at http://127.0.0.1:35861/ using document root <none> and password file <none>
2026-09-29T00:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:40Z Disabled control of system clock
I20260929 00:54:40.164114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34059
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/testchainca.cert with env {}
W20260929 00:54:40.479442 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:40.479810 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:40.479982 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:40.487181 5635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:40.487337 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:40.487435 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:40.487541 5635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:40.487629 5635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:54:40.487720 5635 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:54:40.487799 5635 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45783/jwks.json
I20260929 00:54:40.497761 5635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:40.499068 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:40.500401 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:40.512490 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.512598 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:40.512565 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:40.513985 5635 server_base.cc:1034] running on GCE node
I20260929 00:54:40.514770 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:40.516180 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:40.517406 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790643280517371 us; error 54 us; skew 500 ppm
I20260929 00:54:40.517759 5635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:40.519892 5635 webserver.cc:460] Webserver started at http://127.0.97.62:35853/ using document root <none> and password file <none>
I20260929 00:54:40.520678 5635 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:40.520819 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:40.521152 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:40.523877 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "36bf637f2f3d4afd886ac2cd6ddfba04"
format_stamp: "Formatted at 2026-09-29 00:54:40 on dist-test-slave-pzsx"
server_key: "9c98a5bc39d8057171abad36380f7e72"
server_key_iv: "6b08fab3099ea71e22be5ba1571bd3ec"
server_key_version: "encryptionkey@0"
I20260929 00:54:40.524737 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "36bf637f2f3d4afd886ac2cd6ddfba04"
format_stamp: "Formatted at 2026-09-29 00:54:40 on dist-test-slave-pzsx"
server_key: "9c98a5bc39d8057171abad36380f7e72"
server_key_iv: "6b08fab3099ea71e22be5ba1571bd3ec"
server_key_version: "encryptionkey@0"
I20260929 00:54:40.529644 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:40.533034 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:40.534323 5635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:54:40.534509 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "36bf637f2f3d4afd886ac2cd6ddfba04"
format_stamp: "Formatted at 2026-09-29 00:54:40 on dist-test-slave-pzsx"
server_key: "9c98a5bc39d8057171abad36380f7e72"
server_key_iv: "6b08fab3099ea71e22be5ba1571bd3ec"
server_key_version: "encryptionkey@0"
I20260929 00:54:40.534726 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:40.547441 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:40.548228 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:40.548601 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:40.566342 5635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34059
I20260929 00:54:40.566362 5700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34059 every 8 connection(s)
I20260929 00:54:40.567725 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:40.571401 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:40.571887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5635
I20260929 00:54:40.572072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:40.572713 388 external_mini_cluster.cc:1428] Setting key b6b28f9613f22f5b5b81871c12255458
I20260929 00:54:40.584967 5701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04: Bootstrap starting.
I20260929 00:54:40.586270 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.574631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.97.62:34059)
0929 00:54:40.575628 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:40.575639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:40.575681 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:54:40.577337 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:40.577360 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:40.577816 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:40.578149 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:40.578722 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.579959 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.581881 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.582588 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.582717 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.583956 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:40.583963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:40.583977 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:40.584574 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:40.584644 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:40.584664 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:40.584767 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:40.584980 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 00:54:40.585352 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":393,"threads_started":1}
I20260929 00:54:40.588135 5701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:40.589357 5701 log.cc:824] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:40.589865 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:54:40.592029 5701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04: No bootstrap required, opened a new log
I20260929 00:54:40.599251 5701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260929 00:54:40.599603 5701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:40.599707 5701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bf637f2f3d4afd886ac2cd6ddfba04, State: Initialized, Role: FOLLOWER
I20260929 00:54:40.600334 5701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260929 00:54:40.600620 5701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:40.600762 5701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:40.600919 5701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:40.602887 5701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260929 00:54:40.603335 5701 leader_election.cc:302] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bf637f2f3d4afd886ac2cd6ddfba04; no voters:
I20260929 00:54:40.603899 5701 leader_election.cc:288] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:40.604056 5706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:40.605412 5706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [term 1 LEADER]: Becoming Leader. State: Replica: 36bf637f2f3d4afd886ac2cd6ddfba04, State: Running, Role: LEADER
I20260929 00:54:40.605865 5706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } }
I20260929 00:54:40.605979 5701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:40.608604 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bf637f2f3d4afd886ac2cd6ddfba04. Latest consensus state: current_term: 1 leader_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } } }
I20260929 00:54:40.608659 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf637f2f3d4afd886ac2cd6ddfba04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34059 } } }
I20260929 00:54:40.609004 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:40.609066 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:40.610113 5715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:40.615036 5715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:40.622756 5715 catalog_manager.cc:1347] Generated new cluster ID: 3a1af192dfab405c82a3234fd6fc1d2b
I20260929 00:54:40.622887 5715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:40.632326 5715 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:40.634125 5715 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:40.643548 5715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36bf637f2f3d4afd886ac2cd6ddfba04: Generated new TSK 0
I20260929 00:54:40.644338 5715 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 00:54:40.680855 5731 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 00:54:40.680985 5731 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 00:54:40.682235 5730 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:40.662293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34059 (local address 127.0.0.1:37228)
0929 00:54:40.662920 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:40.662935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:40.663045 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:40.663497 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:40.663500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:40.663508 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:54:40.663769 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.663777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.665181 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.665185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.666233 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:40.666242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.667183 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.667234 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:40.667366 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.667394 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:54:40.681788 (+ 14394us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:54:40.681966 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":347,"threads_started":1}
I20260929 00:54:40.682231 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:40.662435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.97.62:34059)
0929 00:54:40.662582 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:40.662589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:40.663024 (+ 435us) server_negotiation.cc:407] Connection header received
0929 00:54:40.663224 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:40.663230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:40.663322 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:40.663487 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:54:40.663908 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.665054 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.666376 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:40.667070 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:40.667194 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:40.667652 (+ 458us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:54:40.681544 (+ 13892us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 00:54:40.681922 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 00:54:40.682370 5704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 00:54:40.683327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5635
I20260929 00:54:40.684551 5696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:40.773751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5635
2026-09-29T00:54:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 00:54:40.828143 388 mini_oidc.cc:152] Starting JWKS server
I20260929 00:54:40.830017 388 webserver.cc:460] Webserver started at https://127.0.0.1:34209/ using document root <none> and password file <none>
I20260929 00:54:40.830269 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 00:54:40.830860 388 webserver.cc:460] Webserver started at http://127.0.0.1:37063/ using document root <none> and password file <none>
2026-09-29T00:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:40Z Disabled control of system clock
I20260929 00:54:40.855983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34121
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34121
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:54:41.169489 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:41.169780 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:41.169862 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:41.176930 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:41.177039 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:41.177076 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:41.177130 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:41.177166 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:54:41.177245 5738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:54:41.177284 5738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34209/jwks.json
I20260929 00:54:41.186110 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:41.187296 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:41.188985 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:41.200542 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.200486 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.200485 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:41.201509 5738 server_base.cc:1034] running on GCE node
I20260929 00:54:41.202236 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:41.203470 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:41.204713 5738 hybrid_clock.cc:648] HybridClock initialized: now 1790643281204672 us; error 29 us; skew 500 ppm
I20260929 00:54:41.205047 5738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:41.207163 5738 webserver.cc:460] Webserver started at http://127.0.97.62:41515/ using document root <none> and password file <none>
I20260929 00:54:41.207828 5738 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:41.207940 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:41.208276 5738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:41.211093 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "9065041604ad4a3f92bd150aebf24ba4"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "e46ddda0cdd597af13c04c7a963790ac"
server_key_iv: "9e4b4f22fb3fee88dfb0802bea7073a6"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.211879 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "9065041604ad4a3f92bd150aebf24ba4"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "e46ddda0cdd597af13c04c7a963790ac"
server_key_iv: "9e4b4f22fb3fee88dfb0802bea7073a6"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.216827 5738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:54:41.220121 5752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.221431 5738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:41.221606 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "9065041604ad4a3f92bd150aebf24ba4"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "e46ddda0cdd597af13c04c7a963790ac"
server_key_iv: "9e4b4f22fb3fee88dfb0802bea7073a6"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.221824 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:41.257534 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:41.258412 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:41.258713 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:41.277032 5738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34121
I20260929 00:54:41.277042 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34121 every 8 connection(s)
I20260929 00:54:41.278389 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:41.282083 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:41.283581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5738
I20260929 00:54:41.283787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:41.284617 388 external_mini_cluster.cc:1428] Setting key ce47f78ae7ffbd8539ea6650bc1dba86
I20260929 00:54:41.294983 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4: Bootstrap starting.
I20260929 00:54:41.297603 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.286552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.97.62:34121)
0929 00:54:41.287517 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.287531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.287578 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:54:41.289178 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.289208 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.289707 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.290107 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:41.290658 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.291886 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.293290 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.293950 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.294142 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.295333 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:41.295341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:41.295354 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:41.295876 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:41.295961 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:41.295980 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:41.296115 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:41.296341 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 00:54:41.296678 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":340,"threads_started":1}
I20260929 00:54:41.298256 5804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:41.299263 5804 log.cc:824] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:41.301304 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:54:41.302114 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4: No bootstrap required, opened a new log
I20260929 00:54:41.309207 5804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } }
I20260929 00:54:41.309588 5804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:41.309697 5804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9065041604ad4a3f92bd150aebf24ba4, State: Initialized, Role: FOLLOWER
I20260929 00:54:41.310307 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } }
I20260929 00:54:41.310578 5804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:41.310743 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:41.310910 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:41.313076 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } }
I20260929 00:54:41.313563 5804 leader_election.cc:302] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9065041604ad4a3f92bd150aebf24ba4; no voters:
I20260929 00:54:41.314121 5804 leader_election.cc:288] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:41.314247 5809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:41.314953 5809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 9065041604ad4a3f92bd150aebf24ba4, State: Running, Role: LEADER
I20260929 00:54:41.315416 5809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } }
I20260929 00:54:41.316202 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:41.318019 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9065041604ad4a3f92bd150aebf24ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } } }
I20260929 00:54:41.318413 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:41.318329 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9065041604ad4a3f92bd150aebf24ba4. Latest consensus state: current_term: 1 leader_uuid: "9065041604ad4a3f92bd150aebf24ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9065041604ad4a3f92bd150aebf24ba4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34121 } } }
I20260929 00:54:41.318607 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:41.321174 5814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:41.326159 5814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:41.334604 5814 catalog_manager.cc:1347] Generated new cluster ID: 105df28879234c5bb039063579083c61
I20260929 00:54:41.334726 5814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:41.363320 5814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:41.364688 5814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:41.373203 5814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9065041604ad4a3f92bd150aebf24ba4: Generated new TSK 0
I20260929 00:54:41.374011 5814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:54:41.400799 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:41.393055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.97.62:34121)
0929 00:54:41.393318 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:41.393325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:41.393703 (+ 378us) server_negotiation.cc:407] Connection header received
0929 00:54:41.393914 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:41.393918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:41.393982 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:41.394156 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:54:41.394620 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.395385 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.396795 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.397236 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.397382 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.397860 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:54:41.400153 (+ 2293us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:54:41.400461 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 00:54:41.400928 5807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 00:54:41.402884 5833 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:41.392956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34121 (local address 127.0.0.1:46200)
0929 00:54:41.393592 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:41.393613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:41.393761 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:41.394174 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:41.394179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:41.394192 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:54:41.394475 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.394483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.395531 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.395538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.396649 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:41.396661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:41.397365 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:41.397436 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:41.397582 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:41.397615 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:54:41.400372 (+ 2757us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 00:54:41.402406 (+ 2034us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":1758,"thread_start_us":346,"threads_started":1}
I20260929 00:54:41.404067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5738
I20260929 00:54:41.405263 5799 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 00:54:41.412993 5834 webserver.cc:533] Webserver: sslize failed:
E20260929 00:54:41.413089 5834 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 00:54:41.498389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5738
2026-09-29T00:54:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T00:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:41Z Disabled control of system clock
I20260929 00:54:41.578433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38069
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38069
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/testchainca.cert with env {}
W20260929 00:54:41.895205 5839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:41.895526 5839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:41.895617 5839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:41.902616 5839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:41.902729 5839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:41.902765 5839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:41.902812 5839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:41.902856 5839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 00:54:41.902908 5839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 00:54:41.902946 5839 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 00:54:41.912693 5839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:41.913941 5839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:41.915233 5839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:41.929164 5847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.929205 5845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:41.929243 5844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:41.930119 5839 server_base.cc:1034] running on GCE node
I20260929 00:54:41.930917 5839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:41.932641 5839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:41.933871 5839 hybrid_clock.cc:648] HybridClock initialized: now 1790643281933828 us; error 42 us; skew 500 ppm
I20260929 00:54:41.934230 5839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:54:41.940923 5839 webserver.cc:460] Webserver started at http://127.0.97.62:45495/ using document root <none> and password file <none>
I20260929 00:54:41.941900 5839 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:41.942075 5839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:41.942412 5839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:41.945490 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "05245651e5aeda61c7cfa4737fa85942"
server_key_iv: "ffbd03b395bd5057c893978ba2ae0594"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.946472 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "05245651e5aeda61c7cfa4737fa85942"
server_key_iv: "ffbd03b395bd5057c893978ba2ae0594"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.951813 5839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:54:41.955224 5853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:41.956691 5839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:54:41.956874 5839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90"
format_stamp: "Formatted at 2026-09-29 00:54:41 on dist-test-slave-pzsx"
server_key: "05245651e5aeda61c7cfa4737fa85942"
server_key_iv: "ffbd03b395bd5057c893978ba2ae0594"
server_key_version: "encryptionkey@0"
I20260929 00:54:41.957114 5839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:41.980834 5839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:41.981763 5839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:41.982074 5839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:42.000995 5839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38069
I20260929 00:54:42.001044 5904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38069 every 8 connection(s)
I20260929 00:54:42.002398 5839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:42.006012 5905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:42.007575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5839
I20260929 00:54:42.007771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:42.008570 388 external_mini_cluster.cc:1428] Setting key 2f0e7c7bcf84f04bede58e5955827368
I20260929 00:54:42.019799 5905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90: Bootstrap starting.
I20260929 00:54:42.021679 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:42.010445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.97.62:38069)
0929 00:54:42.011452 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:42.011463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:42.011512 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:54:42.013090 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:42.013116 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:42.013570 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:42.013880 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:42.014436 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.015657 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.017456 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.018121 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.018250 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:42.019402 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:42.019409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:42.019422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:54:42.019993 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:42.020067 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:54:42.020093 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:42.020175 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:42.020389 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 00:54:42.020762 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":383,"threads_started":1}
I20260929 00:54:42.023020 5905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:42.024045 5905 log.cc:824] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:42.025327 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:54:42.026876 5905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90: No bootstrap required, opened a new log
I20260929 00:54:42.035935 5905 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } }
I20260929 00:54:42.036363 5905 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:42.036485 5905 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d9f1efd1194dc3b9b0d23e1cf02d90, State: Initialized, Role: FOLLOWER
I20260929 00:54:42.037133 5905 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } }
I20260929 00:54:42.037400 5905 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:42.037528 5905 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:42.037693 5905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:42.040839 5905 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } }
I20260929 00:54:42.041431 5905 leader_election.cc:302] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d9f1efd1194dc3b9b0d23e1cf02d90; no voters:
I20260929 00:54:42.042065 5905 leader_election.cc:288] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:42.042214 5910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:42.043613 5910 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [term 1 LEADER]: Becoming Leader. State: Replica: c5d9f1efd1194dc3b9b0d23e1cf02d90, State: Running, Role: LEADER
I20260929 00:54:42.044374 5910 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } }
I20260929 00:54:42.045205 5905 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:42.047302 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } }
I20260929 00:54:42.047642 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:42.048512 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d9f1efd1194dc3b9b0d23e1cf02d90. Latest consensus state: current_term: 1 leader_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9f1efd1194dc3b9b0d23e1cf02d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } }
I20260929 00:54:42.049866 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:42.054389 5915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:42.060684 5915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:42.071782 5915 catalog_manager.cc:1347] Generated new cluster ID: 4c77451b19dd4dbdaceb5a97bd507d76
I20260929 00:54:42.071935 5915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:42.090118 5915 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:42.091388 5915 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:42.099721 5915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d9f1efd1194dc3b9b0d23e1cf02d90: Generated new TSK 0
I20260929 00:54:42.100797 5915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:54:42.547645 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:42.117785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.0.97.62:38069)
0929 00:54:42.118018 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:42.118028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:42.118460 (+ 432us) server_negotiation.cc:407] Connection header received
0929 00:54:42.118688 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:42.118692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:42.118772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:42.118912 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0929 00:54:42.119555 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.120277 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.122340 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.122767 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.122863 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:42.123404 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 00:54:42.547108 (+423704us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:54:42.547426 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 00:54:42.547829 5908 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 00:54:42.547997 5934 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:42.117724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38069 (local address 127.0.0.1:48124)
0929 00:54:42.118348 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:42.118368 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:42.118531 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:42.118969 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:42.118974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:42.118987 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 00:54:42.119289 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:42.119301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.120453 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.120459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:42.122206 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:42.122221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:42.122895 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:42.122946 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:42.123135 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:42.123182 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 00:54:42.547380 (+424198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 00:54:42.547637 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":317,"threads_started":1}
I20260929 00:54:42.549293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5839
I20260929 00:54:42.550459 5900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:42.650682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5839
2026-09-29T00:54:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1133 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (397 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1465 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:44 dist-test-slave-pzsx krb5kdc[5950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:44 dist-test-slave-pzsx krb5kdc[5950](info): set up 2 sockets
Sep 29 00:54:44 dist-test-slave-pzsx krb5kdc[5950](info): commencing operation
krb5kdc: starting...
W20260929 00:54:45.593328 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:45 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:54:45.787791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44825
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:46.114254 5963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:46.114585 5963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:46.114743 5963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:46.121876 5963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:46.121984 5963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:46.122030 5963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:46.122120 5963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:46.122220 5963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:46.132550 5963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:46.133924 5963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:46.135223 5963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:46.147403 5969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:46.147478 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:46.147570 5968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:46.148350 5963 server_base.cc:1034] running on GCE node
I20260929 00:54:46.149183 5963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:46.150497 5963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:46.151722 5963 hybrid_clock.cc:648] HybridClock initialized: now 1790643286151665 us; error 59 us; skew 500 ppm
I20260929 00:54:46.152060 5963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:46 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:46.155121 5963 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:54:46.156472 5963 webserver.cc:460] Webserver started at http://127.0.97.62:41809/ using document root <none> and password file <none>
I20260929 00:54:46.157187 5963 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:46.157323 5963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:46.157671 5963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:46.160692 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "17d6874e0a7e4705ab9ff03775135b6c"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "41aad9cdd55c346d363ad100b4770ebc"
server_key_iv: "2352e0c5c5f751ebbd1d0ccbd36e5ca7"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.161451 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "17d6874e0a7e4705ab9ff03775135b6c"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "41aad9cdd55c346d363ad100b4770ebc"
server_key_iv: "2352e0c5c5f751ebbd1d0ccbd36e5ca7"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.166381 5963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:46.169636 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:46.170867 5963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:46.171042 5963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "17d6874e0a7e4705ab9ff03775135b6c"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "41aad9cdd55c346d363ad100b4770ebc"
server_key_iv: "2352e0c5c5f751ebbd1d0ccbd36e5ca7"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.171243 5963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:46.189828 5963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:46.193534 5963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:46.193811 5963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:46.211030 5963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44825
I20260929 00:54:46.211076 6029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44825 every 8 connection(s)
I20260929 00:54:46.212340 5963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:46.215919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 5963
I20260929 00:54:46.216002 6030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:46.216142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:46.216816 388 external_mini_cluster.cc:1428] Setting key 6b80f3e7ff761e471c10fb2a9e5d2496
I20260929 00:54:46.229875 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c: Bootstrap starting.
Sep 29 00:54:46 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:46.233451 6030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:46.234508 6030 log.cc:824] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:46.237249 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c: No bootstrap required, opened a new log
I20260929 00:54:46.239900 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:46.218777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.97.62:44825)
0929 00:54:46.219569 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:46.219584 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:46.219632 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:54:46.222171 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:46.222193 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:46.222579 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:46.222905 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:46.224617 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.225835 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.227635 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.228265 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.228383 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:46.232985 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:46.232992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:46.233007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:46.233041 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:46.236011 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:46.236745 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:46.236752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:46.236758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:46.236905 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:46.237271 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:46.237275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:46.237277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:46.237821 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:46.238135 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:46.238530 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 00:54:46.238945 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":337,"threads_started":1}
I20260929 00:54:46.246040 6030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260929 00:54:46.246433 6030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:46.246539 6030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d6874e0a7e4705ab9ff03775135b6c, State: Initialized, Role: FOLLOWER
I20260929 00:54:46.247094 6030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260929 00:54:46.247310 6030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:46.247457 6030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:46.247630 6030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:46.249914 6030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260929 00:54:46.250432 6030 leader_election.cc:302] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17d6874e0a7e4705ab9ff03775135b6c; no voters:
I20260929 00:54:46.251053 6030 leader_election.cc:288] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:46.251175 6035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:46.252275 6035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [term 1 LEADER]: Becoming Leader. State: Replica: 17d6874e0a7e4705ab9ff03775135b6c, State: Running, Role: LEADER
I20260929 00:54:46.252853 6035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } }
I20260929 00:54:46.253258 6030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:46.255457 6036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17d6874e0a7e4705ab9ff03775135b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } } }
I20260929 00:54:46.255915 6036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:46.256184 6037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d6874e0a7e4705ab9ff03775135b6c. Latest consensus state: current_term: 1 leader_uuid: "17d6874e0a7e4705ab9ff03775135b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d6874e0a7e4705ab9ff03775135b6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44825 } } }
I20260929 00:54:46.256584 6037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:46.258853 6044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:46.263753 6044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:46.271941 6044 catalog_manager.cc:1347] Generated new cluster ID: e35bedf22ed74d38b38d53e0e4db4668
I20260929 00:54:46.272045 6044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:46.300789 6044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:46.302158 6044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:46.315176 6044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d6874e0a7e4705ab9ff03775135b6c: Generated new TSK 0
I20260929 00:54:46.316196 6044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:54:46.366739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:46.687908 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:46.688220 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:46.688392 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:46.695369 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:46.695508 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:46.695693 6056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:46.705075 6056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:46.706355 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:46.707655 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:46.720357 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:46.720435 6062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:46.720389 6061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:46.721138 6056 server_base.cc:1034] running on GCE node
I20260929 00:54:46.721709 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:46.722592 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:46.723827 6056 hybrid_clock.cc:648] HybridClock initialized: now 1790643286723785 us; error 54 us; skew 500 ppm
I20260929 00:54:46.724138 6056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:46 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:46.727118 6056 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:54:46.728452 6056 webserver.cc:460] Webserver started at http://127.0.97.1:35133/ using document root <none> and password file <none>
I20260929 00:54:46.729140 6056 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:46.729290 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:46.729580 6056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:46.732391 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a6ba9ed165f4b089e784e072d9c2700"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "179757c158da381797500aee4033ec86"
server_key_iv: "e08a43e164740e5bbda8875a752c3c30"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.733222 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a6ba9ed165f4b089e784e072d9c2700"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "179757c158da381797500aee4033ec86"
server_key_iv: "e08a43e164740e5bbda8875a752c3c30"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.738144 6056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:46.741386 6071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:46.742645 6056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:46.742810 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "4a6ba9ed165f4b089e784e072d9c2700"
format_stamp: "Formatted at 2026-09-29 00:54:46 on dist-test-slave-pzsx"
server_key: "179757c158da381797500aee4033ec86"
server_key_iv: "e08a43e164740e5bbda8875a752c3c30"
server_key_version: "encryptionkey@0"
I20260929 00:54:46.743005 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:46.769918 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:46.773712 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:46.774003 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:46.774983 6056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:46.776417 6056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:46.776540 6056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:46.776690 6056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:46.776767 6056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:46.808632 6056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34687
I20260929 00:54:46.808682 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34687 every 8 connection(s)
I20260929 00:54:46.809902 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:46.817222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6056
I20260929 00:54:46.817431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:46.818190 388 external_mini_cluster.cc:1428] Setting key 3dbd7deb72f0123dbd7a20c46a19c6ac
Sep 29 00:54:46 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:46.829588 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:46.812373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33501 (local address 127.0.97.62:44825)
0929 00:54:46.812819 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:46.812824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:46.813791 (+ 967us) server_negotiation.cc:407] Connection header received
0929 00:54:46.815556 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:46.815560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:46.815623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:46.815717 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:46.817184 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.818383 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.819880 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.820278 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.820413 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:46.823985 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:46.823990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:46.823993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:46.824023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:46.826517 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:46.827245 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:46.827249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:46.827252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:46.827372 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:46.827809 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:46.827816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:46.827824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:46.828077 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:46.828240 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:46.829006 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 00:54:46.829232 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260929 00:54:46.830534 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:46.812926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.1:33501)
0929 00:54:46.813628 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:46.813672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:46.815270 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:46.815911 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:46.815919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:46.816411 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:46.817015 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:46.817032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.818527 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.818531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:46.819736 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:46.819748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:46.820516 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:46.820525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:46.820751 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:46.821437 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:46.821457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:46.823769 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:46.826660 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:46.826669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:46.826685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:46.827103 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:46.827482 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:46.827487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:46.827491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:46.827681 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:46.828205 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:46.828212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:46.828608 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 00:54:46.828881 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 00:54:46.829429 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":273,"threads_started":1}
I20260929 00:54:46.832056 6184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260929 00:54:46.832388 6184 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:46.833300 6184 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260929 00:54:46.835520 5995 ts_manager.cc:194] Registered new tserver with Master: 4a6ba9ed165f4b089e784e072d9c2700 (127.0.97.1:34687)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:46.836998 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:54:46.863219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:47.183801 6190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:47.184057 6190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:47.184208 6190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:47.191234 6190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:47.191330 6190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:47.191484 6190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:47.200870 6190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:47.202088 6190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:47.203343 6190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:47.216370 6196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:47.216989 6198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:47.217015 6195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:47.218236 6190 server_base.cc:1034] running on GCE node
I20260929 00:54:47.218933 6190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:47.220093 6190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:47.221309 6190 hybrid_clock.cc:648] HybridClock initialized: now 1790643287221257 us; error 43 us; skew 500 ppm
I20260929 00:54:47.221658 6190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.231031 6190 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:54:47.232605 6190 webserver.cc:460] Webserver started at http://127.0.97.2:44709/ using document root <none> and password file <none>
I20260929 00:54:47.233328 6190 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:47.233464 6190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:47.233794 6190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:47.236879 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "4011c0a7145944249bbd54994854ed02"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "e3a4bf679ee6904e7c304ba351f39357"
server_key_iv: "27e618be902c7facefea6f056b4dfe0f"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.237672 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4011c0a7145944249bbd54994854ed02"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "e3a4bf679ee6904e7c304ba351f39357"
server_key_iv: "27e618be902c7facefea6f056b4dfe0f"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.242929 6190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:47.246392 6205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.247807 6190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:47.247977 6190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "4011c0a7145944249bbd54994854ed02"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "e3a4bf679ee6904e7c304ba351f39357"
server_key_iv: "27e618be902c7facefea6f056b4dfe0f"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.248181 6190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:47.266395 6190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:47.270402 6190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:47.270684 6190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:47.271661 6190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:47.272996 6190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:47.273100 6190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.273257 6190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:47.273330 6190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.305547 6190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45521
I20260929 00:54:47.305583 6317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45521 every 8 connection(s)
I20260929 00:54:47.306794 6190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:47.312441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6190
I20260929 00:54:47.312649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:47.313058 388 external_mini_cluster.cc:1428] Setting key c98e954db4ccba64561a61897bd9b97d
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:47.328318 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.309274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58379 (local address 127.0.97.62:44825)
0929 00:54:47.309477 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.309483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.310557 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 00:54:47.312384 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.312389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.312491 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.312670 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.314224 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.315122 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.317644 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.318049 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.318152 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.322598 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.322603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.322607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.322636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.325157 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.326064 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.326067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.326069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.326187 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.326654 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.326656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.326658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.326911 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.327029 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.327914 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.328068 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:54:47.330001 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.309723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.2:58379)
0929 00:54:47.310408 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:47.310445 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:47.312117 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:47.312911 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:47.312920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:47.313405 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:47.314057 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:47.314074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.315295 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.315300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:47.317503 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:47.317520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.318272 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.318281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:47.318620 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.319351 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:47.319386 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:47.322350 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:47.325296 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:47.325308 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:47.325328 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:47.325956 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:47.326295 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:47.326303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:47.326310 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:47.326552 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:47.327051 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:47.327062 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:47.327527 (+ 465us) client_negotiation.cc:769] Sending connection context
0929 00:54:47.327854 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 00:54:47.328431 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":282,"threads_started":1}
I20260929 00:54:47.331525 6318 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260929 00:54:47.331827 6318 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:47.332602 6318 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260929 00:54:47.334031 5995 ts_manager.cc:194] Registered new tserver with Master: 4011c0a7145944249bbd54994854ed02 (127.0.97.2:45521)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:54:47.334864 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:54:47.360249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:47.674569 6323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:47.674834 6323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:47.675010 6323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:47.682428 6323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:47.682528 6323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:47.682691 6323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:47.692080 6323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:47.693349 6323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:47.694615 6323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:47.707317 6328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:47.707283 6329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:47.707239 6331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:47.708051 6323 server_base.cc:1034] running on GCE node
I20260929 00:54:47.708722 6323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:47.709656 6323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:47.710886 6323 hybrid_clock.cc:648] HybridClock initialized: now 1790643287710847 us; error 52 us; skew 500 ppm
I20260929 00:54:47.711189 6323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.714289 6323 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:54:47.715552 6323 webserver.cc:460] Webserver started at http://127.0.97.3:42131/ using document root <none> and password file <none>
I20260929 00:54:47.716173 6323 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:47.716305 6323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:47.716662 6323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:47.719514 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "b031a92b5d3f48abb2f09e7ed52818fa"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "aa1c93a7dfa714391d31134417c6a444"
server_key_iv: "f59e0e4f682a5e8c68d3d3185c2b89ad"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.720265 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b031a92b5d3f48abb2f09e7ed52818fa"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "aa1c93a7dfa714391d31134417c6a444"
server_key_iv: "f59e0e4f682a5e8c68d3d3185c2b89ad"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.725080 6323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:54:47.728322 6338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.729563 6323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:47.729725 6323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "b031a92b5d3f48abb2f09e7ed52818fa"
format_stamp: "Formatted at 2026-09-29 00:54:47 on dist-test-slave-pzsx"
server_key: "aa1c93a7dfa714391d31134417c6a444"
server_key_iv: "f59e0e4f682a5e8c68d3d3185c2b89ad"
server_key_version: "encryptionkey@0"
I20260929 00:54:47.729914 6323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:47.742836 6323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:47.747320 6323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:47.747615 6323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:47.748564 6323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:47.749974 6323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:47.750087 6323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.750208 6323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:47.750258 6323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:47.782956 6323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43879
I20260929 00:54:47.782998 6450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43879 every 8 connection(s)
I20260929 00:54:47.784228 6323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:47.788380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6323
I20260929 00:54:47.788580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:47.789016 388 external_mini_cluster.cc:1428] Setting key 8036b98df58d3e13371b396e3dec8e6e
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:54:47.805147 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.786742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51289 (local address 127.0.97.62:44825)
0929 00:54:47.786903 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.786907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.788173 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 00:54:47.790966 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.790969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.791032 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.791109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.793189 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.793989 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.796191 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.796618 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.796719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.799802 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.799807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.799810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.799840 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.802120 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.802890 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.802902 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.802905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.803031 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.803450 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.803456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.803460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.803785 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.803936 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.804613 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.804819 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:54:47.806017 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.787156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44825 (local address 127.0.97.3:51289)
0929 00:54:47.787993 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:47.788051 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:47.790663 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:47.791372 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:47.791383 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:47.792152 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:47.793006 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:47.793036 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.794155 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.794163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:47.796068 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:47.796080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.796793 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.796798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:47.796975 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.797576 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:47.797597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:47.799623 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:47.802269 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:47.802278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:47.802293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:47.802761 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:47.803150 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:47.803154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:47.803158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:47.803336 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:47.803913 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:47.803921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:47.804255 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 00:54:47.804502 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 00:54:47.805013 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":311,"threads_started":1}
I20260929 00:54:47.807178 6451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44825
I20260929 00:54:47.807500 6451 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:47.808211 6451 heartbeater.cc:511] Master 127.0.97.62:44825 requested a full tablet report, sending...
I20260929 00:54:47.809669 5995 ts_manager.cc:194] Registered new tserver with Master: b031a92b5d3f48abb2f09e7ed52818fa (127.0.97.3:43879)
I20260929 00:54:47.810753 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51289
I20260929 00:54:47.815810 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/test-admin.keytab.
I20260929 00:54:47.839898 6184 heartbeater.cc:503] Master 127.0.97.62:44825 was elected leader, sending a full tablet report...
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.842078 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 00:54:47.847033 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:47.844599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.0.97.1:34687)
0929 00:54:47.845196 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.845203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.845230 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:54:47.845283 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.845289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.845501 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.845640 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.846601 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43232: BlockingRecv error: recv got EOF from 127.0.0.1:43232 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
W20260929 00:54:47.847170 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.849071 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.850472 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:47.866299 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.852172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.97.1:34687)
0929 00:54:47.852374 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.852383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.852661 (+ 278us) server_negotiation.cc:407] Connection header received
0929 00:54:47.852869 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.852876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.852940 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.853042 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.854098 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.855115 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.856445 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.857151 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.857309 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.860821 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.860831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.860847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.860885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.863513 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.864163 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.864171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.864173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.864291 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.864753 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.864765 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.864768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.865089 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.865395 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.865735 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.865977 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 00:54:47.871816 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:47.870165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.97.1:34687)
0929 00:54:47.870362 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.870368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.870534 (+ 166us) server_negotiation.cc:407] Connection header received
0929 00:54:47.870718 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.870723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.870789 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.870898 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.871542 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43250: BlockingRecv error: recv got EOF from 127.0.0.1:43250 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 00:54:47.871987 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.873469 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.874828 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:47.888273 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.876007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.0.97.1:34687)
0929 00:54:47.876211 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.876220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.876471 (+ 251us) server_negotiation.cc:407] Connection header received
0929 00:54:47.876692 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.876697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.876761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.876857 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.877764 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.878553 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.879717 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.880142 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.880239 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.883854 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.883858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.883861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.883888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.886197 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.886779 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.886782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.886784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.886891 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.887175 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.887178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.887180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.887406 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.887581 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.887801 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.887939 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 00:54:47.892156 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:54:47.890658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.97.1:34687)
0929 00:54:47.890861 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.890867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.891043 (+ 176us) server_negotiation.cc:407] Connection header received
0929 00:54:47.891154 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.891157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.891223 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.891297 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.891946 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43266: BlockingRecv error: recv got EOF from 127.0.0.1:43266 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 00:54:47.892294 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643237839665-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.893792 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:47.894994 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 00:54:47 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:54:47.907339 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.896117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.0.97.1:34687)
0929 00:54:47.896251 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.896254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.896658 (+ 404us) server_negotiation.cc:407] Connection header received
0929 00:54:47.896749 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.896752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.896800 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.896872 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.897713 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.898392 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.899552 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.899964 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.900055 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.903147 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.903152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.903155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.903182 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.905427 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.905956 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.905959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.905962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.906092 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.906369 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.906372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.906374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.906605 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.906827 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.906952 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.907091 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:54:47.908989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6056
I20260929 00:54:47.919816 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:47.909656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.0.97.1:34687)
0929 00:54:47.909789 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:47.909793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:47.910198 (+ 405us) server_negotiation.cc:407] Connection header received
0929 00:54:47.910298 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:47.910301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:47.910352 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:47.910421 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:47.911228 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.911853 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.913055 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:47.913457 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:47.913544 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:47.915348 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:47.915353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:47.915356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:47.915382 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:47.917819 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.918381 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.918385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.918387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.918508 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:47.918795 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:47.918798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:47.918800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:47.919015 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:47.919203 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:47.919414 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 00:54:47.919561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:54:47.920344 6179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:48.002864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6056
I20260929 00:54:48.018939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6190
Sep 29 00:54:48 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:54:48.034190 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:48.019731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.97.2:45521)
0929 00:54:48.020259 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:48.020266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:48.020293 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:54:48.020352 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:48.020357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:48.020629 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:48.020767 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:48.021634 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:48.022576 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:48.023716 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:48.024235 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:48.024334 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:48.027988 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:48.027999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:48.028012 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:48.028043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:48.030784 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:48.031452 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:48.031463 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:48.031469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:48.031620 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:48.032090 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:48.032097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:48.032102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:48.032604 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:48.033008 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:48.033367 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 00:54:48.033664 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260929 00:54:48.035950 6313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:48.118587 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6190
I20260929 00:54:48.134114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6323
Sep 29 00:54:48 dist-test-slave-pzsx krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:54:48.149164 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:48.135137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.97.3:43879)
0929 00:54:48.135771 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:48.135779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:48.135806 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:54:48.135861 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:48.135868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:48.136075 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:48.136208 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:48.137237 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:48.138242 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:48.139454 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:48.139997 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:48.140090 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:48.143856 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:48.143867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:48.143881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:48.143913 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:48.146830 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:48.147425 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:48.147432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:48.147434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:48.147538 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:48.147846 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:48.147849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:48.147851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:48.148194 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:48.148422 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:48.148677 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 00:54:48.148860 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":347,"threads_started":1}
I20260929 00:54:48.150470 6446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:48.232607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6323
I20260929 00:54:48.248798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 5963
I20260929 00:54:48.249972 6025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:48.351516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 5963
2026-09-29T00:54:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3842 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:48 dist-test-slave-pzsx krb5kdc[6478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:48 dist-test-slave-pzsx krb5kdc[6478](info): set up 2 sockets
Sep 29 00:54:48 dist-test-slave-pzsx krb5kdc[6478](info): commencing operation
krb5kdc: starting...
W20260929 00:54:49.462208 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:54:49 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:54:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 00:54:49.660780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46767
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:49.974212 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:49.974484 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:49.974653 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:49.981571 6491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:49.981674 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:49.981769 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:49.981865 6491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:49.981925 6491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:49.991328 6491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:49.992606 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:49.993893 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:50.005839 6496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:50.005805 6497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:50.005761 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:50.006758 6491 server_base.cc:1034] running on GCE node
I20260929 00:54:50.007542 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:50.009043 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:50.010277 6491 hybrid_clock.cc:648] HybridClock initialized: now 1790643290010228 us; error 60 us; skew 500 ppm
I20260929 00:54:50.010655 6491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:50 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:50.013695 6491 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 00:54:50.015089 6491 webserver.cc:460] Webserver started at http://127.0.97.62:40429/ using document root <none> and password file <none>
I20260929 00:54:50.015789 6491 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:50.015931 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:50.016234 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:50.019224 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "bfad066ad17245efbe1c29c65730bb11"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "9539ba3715e2c67962312638c5fbd910"
server_key_iv: "a51e37d043391539fffaf9eb62aa87d6"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.019969 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "bfad066ad17245efbe1c29c65730bb11"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "9539ba3715e2c67962312638c5fbd910"
server_key_iv: "a51e37d043391539fffaf9eb62aa87d6"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.025010 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 00:54:50.028308 6506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:50.029686 6491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:54:50.029861 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "bfad066ad17245efbe1c29c65730bb11"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "9539ba3715e2c67962312638c5fbd910"
server_key_iv: "a51e37d043391539fffaf9eb62aa87d6"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.030048 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:50.065891 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:50.070291 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:50.070581 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:50.088550 6491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46767
I20260929 00:54:50.088569 6557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46767 every 8 connection(s)
I20260929 00:54:50.089979 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:54:50.090315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6491
I20260929 00:54:50.090510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:54:50.091359 388 external_mini_cluster.cc:1428] Setting key bf13901d3fc8ec53481b0c12efd1f33a
I20260929 00:54:50.094703 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 00:54:50 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:50.107476 6558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Bootstrap starting.
I20260929 00:54:50.111605 6558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:50.112974 6558 log.cc:824] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:50.115310 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:50.093412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.97.62:46767)
0929 00:54:50.094219 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:50.094236 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:50.094289 (+ 53us) server_negotiation.cc:407] Connection header received
0929 00:54:50.096475 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:50.096528 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:50.097024 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:50.097437 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:50.098707 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.100723 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.101842 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.102875 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.103056 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:50.107080 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:50.107093 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:50.107112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:50.107159 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:50.110957 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:50.111595 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:50.111605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:50.111613 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:50.111811 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:50.112162 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:50.112184 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:50.112187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:50.112893 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:50.113184 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:50.113639 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 00:54:50.114104 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":335,"threads_started":1}
I20260929 00:54:50.116240 6558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: No bootstrap required, opened a new log
I20260929 00:54:50.123670 6558 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } }
I20260929 00:54:50.123991 6558 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:50.124070 6558 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad066ad17245efbe1c29c65730bb11, State: Initialized, Role: FOLLOWER
I20260929 00:54:50.124660 6558 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } }
I20260929 00:54:50.124960 6558 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:54:50.125126 6558 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:54:50.125326 6558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:50.127364 6558 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } }
I20260929 00:54:50.127835 6558 leader_election.cc:302] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfad066ad17245efbe1c29c65730bb11; no voters:
I20260929 00:54:50.128422 6558 leader_election.cc:288] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:54:50.128577 6563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:50.129874 6563 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 LEADER]: Becoming Leader. State: Replica: bfad066ad17245efbe1c29c65730bb11, State: Running, Role: LEADER
I20260929 00:54:50.130365 6563 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } }
I20260929 00:54:50.130652 6558 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:50.133132 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } }
I20260929 00:54:50.133091 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfad066ad17245efbe1c29c65730bb11. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } }
I20260929 00:54:50.133529 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:50.133534 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:50.135324 6572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:50.140030 6572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:50.148663 6572 catalog_manager.cc:1347] Generated new cluster ID: 92e065524b9d4501af31b5d644ca9324
I20260929 00:54:50.148778 6572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:50.164304 6572 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:54:50.166163 6572 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:50.180419 6572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Generated new TSK 0
I20260929 00:54:50.181479 6572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 00:54:50.232384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:50.553791 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:50.554100 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:50.554229 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:50.561486 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:50.561599 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:50.561726 6584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:50.571350 6584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:50.572626 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:50.573900 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:50.586524 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:50.586505 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:50.586527 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:50.587157 6584 server_base.cc:1034] running on GCE node
I20260929 00:54:50.587778 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:50.588750 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:50.589938 6584 hybrid_clock.cc:648] HybridClock initialized: now 1790643290589892 us; error 38 us; skew 500 ppm
I20260929 00:54:50.590234 6584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:50 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:50.593330 6584 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 00:54:50.594714 6584 webserver.cc:460] Webserver started at http://127.0.97.1:46069/ using document root <none> and password file <none>
I20260929 00:54:50.595362 6584 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:50.595487 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:50.595794 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:50.598655 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "26f9db9faab8437e8608c109ee3e9837"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "27ad0d37738312eea344bc04cfab82a3"
server_key_iv: "f58cd4d1028d0a144a688c0b76a324c5"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.599406 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "26f9db9faab8437e8608c109ee3e9837"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "27ad0d37738312eea344bc04cfab82a3"
server_key_iv: "f58cd4d1028d0a144a688c0b76a324c5"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.604241 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260929 00:54:50.607450 6599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:50.608716 6584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:50.608863 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "26f9db9faab8437e8608c109ee3e9837"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "27ad0d37738312eea344bc04cfab82a3"
server_key_iv: "f58cd4d1028d0a144a688c0b76a324c5"
server_key_version: "encryptionkey@0"
I20260929 00:54:50.609046 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:50.626709 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:50.631032 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:50.631305 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:50.632316 6584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:50.633646 6584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:50.633744 6584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:50.633885 6584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:50.633955 6584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:50.666675 6584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43427
I20260929 00:54:50.666720 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43427 every 8 connection(s)
I20260929 00:54:50.667984 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:50.671875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6584
I20260929 00:54:50.672070 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:54:50.672953 388 external_mini_cluster.cc:1428] Setting key 0d87271d59a938c4896e962ee581a889
Sep 29 00:54:50 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:50.690717 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:50.670405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39005 (local address 127.0.97.62:46767)
0929 00:54:50.670865 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:50.670871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:50.671926 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 00:54:50.674302 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:50.674306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:50.674373 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:50.674466 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:50.676399 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.677130 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.679643 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.680145 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.680280 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:50.684034 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:50.684042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:50.684050 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:50.684091 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:50.687568 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:50.688338 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:50.688346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:50.688353 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:50.688522 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:50.688925 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:50.688932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:50.688939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:50.689341 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:50.689482 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:50.690163 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 00:54:50.690371 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260929 00:54:50.691759 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:50.670949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:39005)
0929 00:54:50.671733 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:50.671793 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:50.673997 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:50.674731 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:50.674739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:50.675426 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:50.676228 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:50.676254 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.677278 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.677286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:50.679362 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:50.679378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:50.680394 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:50.680403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:50.680756 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:50.681465 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:50.681500 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:50.683818 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:50.687796 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:50.687803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:50.687816 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:50.688213 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:50.688636 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:50.688640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:50.688642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:50.688806 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:50.689464 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:50.689470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:50.689805 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 00:54:50.690078 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 00:54:50.690588 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":263,"threads_started":1}
I20260929 00:54:50.693310 6712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:50.693648 6712 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:50.694567 6712 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:54:50.696830 6523 ts_manager.cc:194] Registered new tserver with Master: 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427)
I20260929 00:54:50.698189 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 00:54:50.722000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:51.048923 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:51.049233 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:51.049441 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:51.056619 6718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:51.056730 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:51.056898 6718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:51.066623 6718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:51.068007 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:51.069866 6718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:51.082898 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:51.082942 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:51.082973 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:51.083729 6718 server_base.cc:1034] running on GCE node
I20260929 00:54:51.084371 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:51.085342 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:51.086565 6718 hybrid_clock.cc:648] HybridClock initialized: now 1790643291086524 us; error 56 us; skew 500 ppm
I20260929 00:54:51.086880 6718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:51 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643291, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:51.090174 6718 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 00:54:51.092581 6718 webserver.cc:460] Webserver started at http://127.0.97.2:39903/ using document root <none> and password file <none>
I20260929 00:54:51.093263 6718 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:51.093406 6718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:51.093709 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:51.096587 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "87284e1bb781498db54719968ba55cd8"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "0c3292a992cb1926d80208b23e82dd6a"
server_key_iv: "a33ad42d6da09c98364ef739b75c84d7"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.097371 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87284e1bb781498db54719968ba55cd8"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "0c3292a992cb1926d80208b23e82dd6a"
server_key_iv: "a33ad42d6da09c98364ef739b75c84d7"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.102599 6718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:54:51.105986 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.107282 6718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:54:51.107448 6718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "87284e1bb781498db54719968ba55cd8"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "0c3292a992cb1926d80208b23e82dd6a"
server_key_iv: "a33ad42d6da09c98364ef739b75c84d7"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.107650 6718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:51.130577 6718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:51.135210 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:51.135486 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:51.136474 6718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:51.137789 6718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:51.137907 6718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.138047 6718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:51.138115 6718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.171100 6718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45467
I20260929 00:54:51.171128 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45467 every 8 connection(s)
I20260929 00:54:51.172384 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:51.181538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6718
I20260929 00:54:51.181726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:54:51.182174 388 external_mini_cluster.cc:1428] Setting key 2618b883b8e1330cf228229814a8f740
Sep 29 00:54:51 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643291, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:51.192227 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:51.174936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60677 (local address 127.0.97.62:46767)
0929 00:54:51.175133 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:51.175139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:51.176371 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 00:54:51.178131 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:51.178135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:51.178200 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:51.178289 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:51.179689 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.180394 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.182181 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.182763 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.182901 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:51.186802 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:51.186807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:51.186810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:51.186839 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:51.189379 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:51.190065 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:51.190074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:51.190077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:51.190195 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:51.190602 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:51.190606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:51.190610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:51.190851 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:51.190991 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:51.191670 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 00:54:51.191882 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:54:51.193279 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:51.175505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:60677)
0929 00:54:51.176198 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:51.176250 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:51.177890 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:51.178487 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:51.178495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:51.178993 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:51.179547 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:51.179568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.180589 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.180593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:51.182042 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:51.182053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.182986 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.182993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:51.183296 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:51.184062 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:51.184122 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:51.186571 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:51.189513 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:51.189522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:51.189539 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:51.189941 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:51.190300 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:51.190306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:51.190310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:51.190489 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:51.190981 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:51.190988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:51.191325 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 00:54:51.191578 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 00:54:51.192086 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260929 00:54:51.194692 6846 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:51.195011 6846 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:51.195847 6846 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
I20260929 00:54:51.197418 6523 ts_manager.cc:194] Registered new tserver with Master: 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467)
I20260929 00:54:51.198197 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60677
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 00:54:51.227742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:51.544217 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:51.544488 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:51.544653 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:51.551756 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:51.551862 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:51.551976 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:51.561522 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:51.562688 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:51.563885 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:51.576442 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:51.576530 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:51.576536 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:51.577296 6851 server_base.cc:1034] running on GCE node
I20260929 00:54:51.577893 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:51.578753 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:51.579969 6851 hybrid_clock.cc:648] HybridClock initialized: now 1790643291579915 us; error 51 us; skew 500 ppm
I20260929 00:54:51.580272 6851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:51 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643291, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:51.583261 6851 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 00:54:51.584653 6851 webserver.cc:460] Webserver started at http://127.0.97.3:40577/ using document root <none> and password file <none>
I20260929 00:54:51.585305 6851 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:51.585448 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:51.585744 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:51.588572 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "e4a3178fdc8c49b2a208caad63034b98"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "a9fd4b41ffe4afeb7be840e63d62fb19"
server_key_iv: "86c7fe5049029de93dfb643e2fc65059"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.589465 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e4a3178fdc8c49b2a208caad63034b98"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "a9fd4b41ffe4afeb7be840e63d62fb19"
server_key_iv: "86c7fe5049029de93dfb643e2fc65059"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.594234 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 00:54:51.597436 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.598632 6851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 00:54:51.598801 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "e4a3178fdc8c49b2a208caad63034b98"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "a9fd4b41ffe4afeb7be840e63d62fb19"
server_key_iv: "86c7fe5049029de93dfb643e2fc65059"
server_key_version: "encryptionkey@0"
I20260929 00:54:51.598984 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:51.609146 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:51.613041 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:51.613379 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:51.614248 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:51.615506 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:51.615617 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.615764 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:51.615836 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:51.647886 6851 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35065
I20260929 00:54:51.647933 6978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35065 every 8 connection(s)
I20260929 00:54:51.649240 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:51.657805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6851
I20260929 00:54:51.657994 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:54:51.658444 388 external_mini_cluster.cc:1428] Setting key 83d7616bd5ce85c151c26acc1748d133
Sep 29 00:54:51 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643291, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:51.668848 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:51.651659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58911 (local address 127.0.97.62:46767)
0929 00:54:51.651840 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:51.651844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:51.653069 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 00:54:51.654831 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:51.654837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:51.654921 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:51.655028 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:51.656461 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.657266 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.658944 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.659367 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.659498 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:51.663234 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:51.663238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:51.663241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:51.663270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:51.665795 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:51.666640 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:51.666644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:51.666646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:51.666762 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:51.667266 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:51.667269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:51.667271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:51.667506 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:51.667617 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:51.668404 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 00:54:51.668617 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:54:51.670149 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:51.652180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:58911)
0929 00:54:51.652912 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:51.652951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:51.654564 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:51.655263 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:51.655272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:51.655766 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:51.656300 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:51.656316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.657412 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.657423 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:51.658770 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:51.658785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:51.659609 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:51.659617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:51.659950 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:51.660766 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:51.660787 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:51.663046 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:51.665931 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:51.665940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:51.665956 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:51.666529 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:51.666890 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:51.666898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:51.666901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:51.667125 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:51.667628 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:51.667636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:51.668033 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 00:54:51.668337 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 00:54:51.668863 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":317,"threads_started":1}
I20260929 00:54:51.671511 6979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:51.671824 6979 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:51.672626 6979 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
I20260929 00:54:51.673943 6523 ts_manager.cc:194] Registered new tserver with Master: e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:51.674669 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58911
I20260929 00:54:51.685437 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:54:51.701102 6712 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 00:54:51.729905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34477
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46767,127.0.97.61:34477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:52.059186 6984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:52.059532 6984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:52.059687 6984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:52.067013 6984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:52.067165 6984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:52.067273 6984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:52.067375 6984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:52.067448 6984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:52.078234 6984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46767,127.0.97.61:34477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
W20260929 00:54:52.078599 6984 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46767,127.0.97.61:34477'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 00:54:52.079669 6984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:52.081082 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:52.093218 6989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:52.093333 6990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:52.093842 6992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:52.095172 6984 server_base.cc:1034] running on GCE node
I20260929 00:54:52.095942 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:52.097396 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:52.098632 6984 hybrid_clock.cc:648] HybridClock initialized: now 1790643292098571 us; error 40 us; skew 500 ppm
I20260929 00:54:52.098956 6984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:52.102344 6984 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 00:54:52.103804 6984 webserver.cc:460] Webserver started at http://127.0.97.61:33997/ using document root <none> and password file <none>
I20260929 00:54:52.104475 6984 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:52.104653 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:52.104997 6984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:52.107842 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/instance:
uuid: "05a723e6d0384131b6a81a2520b9d979"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "380b9167eb55e52abdcea073204a6c30"
server_key_iv: "7b5156c9bc766aac94e9cf65c6254d86"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.108688 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal/instance:
uuid: "05a723e6d0384131b6a81a2520b9d979"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "380b9167eb55e52abdcea073204a6c30"
server_key_iv: "7b5156c9bc766aac94e9cf65c6254d86"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.113751 6984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:54:52.117192 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:52.118476 6984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:54:52.118649 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "05a723e6d0384131b6a81a2520b9d979"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "380b9167eb55e52abdcea073204a6c30"
server_key_iv: "7b5156c9bc766aac94e9cf65c6254d86"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.118850 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:52.150198 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:52.154824 6984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:52.155097 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:52.173743 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34477
I20260929 00:54:52.173712 7050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34477 every 8 connection(s)
I20260929 00:54:52.175386 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
I20260929 00:54:52.179153 7051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:52.180874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 6984
I20260929 00:54:52.188555 7051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:52.201521 6846 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
I20260929 00:54:52.205425 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.183040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.0.97.61:34477)
0929 00:54:52.183924 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.183935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.183974 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:54:52.185571 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.185597 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.185994 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.186312 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.188074 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.189464 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.191087 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.192214 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.192387 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.197185 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.197191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.197202 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.197237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.200769 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.201501 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.201516 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.201523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.201724 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.202148 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.202155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.202162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.202713 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.203021 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.203522 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.203968 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":443,"threads_started":1}
I20260929 00:54:52.207998 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.191246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.0.97.62:46767)
0929 00:54:52.191767 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.191773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.192338 (+ 565us) server_negotiation.cc:407] Connection header received
0929 00:54:52.192488 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.192494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.192601 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.192714 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.193924 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.195119 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.196806 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.197326 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.197415 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.201289 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.201295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.201299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.201336 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.205453 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.206204 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.206209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.206212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.206370 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.206690 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.206694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.206697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.207001 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.207113 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.207637 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.207782 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260929 00:54:52.208317 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.191302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39672)
0929 00:54:52.192162 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.192221 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.192369 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.192831 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.192835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.193534 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.193770 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.193778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.195244 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.195247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.196672 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.196684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.197502 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.197508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.197710 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.198343 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.198371 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.201078 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.205599 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.205613 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.205623 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.206095 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.206483 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.206486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.206488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.206593 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.207140 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.207150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.207336 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.207569 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.207872 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":477,"threads_started":1}
I20260929 00:54:52.209897 7051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 00:54:52.221261 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.210640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57282)
0929 00:54:52.210832 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.210847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.210922 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.211174 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.211177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.211454 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.211575 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.211582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.212569 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.212574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.213433 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.213443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.214091 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.214096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.214306 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.214858 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.214873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.216815 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.219297 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.219300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.219303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.219635 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.219916 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.219919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.219921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.220035 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.220475 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.220478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.220670 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.220784 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.220983 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":6}
I20260929 00:54:52.221338 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.210837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.0.97.61:34477)
0929 00:54:52.210968 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.210972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.210986 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:54:52.211020 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.211023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.211070 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.211143 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.211701 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.212425 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.213568 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.213950 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.214083 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.216971 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.216976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.216979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.217005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.219187 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.219727 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.219730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.219732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.219828 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.220125 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.220128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.220130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.220385 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.220484 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.220867 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.221093 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":42}
I20260929 00:54:52.226262 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Bootstrap starting.
I20260929 00:54:52.229377 7051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:52.230384 7051 log.cc:824] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:52.233130 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: No bootstrap required, opened a new log
I20260929 00:54:52.240101 7051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } }
I20260929 00:54:52.240412 7051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:52.240485 7051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a723e6d0384131b6a81a2520b9d979, State: Initialized, Role: FOLLOWER
I20260929 00:54:52.241070 7051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } }
I20260929 00:54:52.241830 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } } }
I20260929 00:54:52.242152 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:52.242806 7051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:54:52.251075 7069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:54:52.251221 7069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:54:52.253981 6984 master_runner.cc:428] Detected that this master 05a723e6d0384131b6a81a2520b9d979 is joining an existing cluster
I20260929 00:54:52.254093 6984 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34477
I20260929 00:54:52.278980 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.257806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39680)
0929 00:54:52.258762 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.258789 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.258977 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.259581 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.259589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.260407 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.261232 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.261244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.262623 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.262633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.264542 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.264557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.265808 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.265820 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.266079 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.270322 (+ 4243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.270340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.271282 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.275916 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.275922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.275929 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.276567 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.276985 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.276991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.276995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.277160 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.277796 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.277803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.278007 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.278154 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.278473 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":213,"spinlock_wait_cycles":24832,"thread_start_us":2664,"threads_started":1}
I20260929 00:54:52.279446 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.258231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.97.61:34477)
0929 00:54:52.258399 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.258405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.259479 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 00:54:52.259662 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.259669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.259751 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.259813 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.261383 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.262454 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.264687 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.265434 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.265511 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.273022 (+ 7511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.273028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.273035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.273068 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.276679 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.277433 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.277438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.277441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.277579 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.277994 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.277998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.278001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.278316 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.278467 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.278902 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.279112 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 00:54:52.280285 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.257829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.97.62:46767)
0929 00:54:52.258009 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.258016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.258872 (+ 856us) server_negotiation.cc:407] Connection header received
0929 00:54:52.259164 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.259171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.259293 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.259367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.261397 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.262454 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.264700 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.265394 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.265500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.271483 (+ 5983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.271491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.271497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.271538 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.275757 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.276700 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.276706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.276709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.276861 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.277300 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.277304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.277307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.277674 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.279497 (+ 1823us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.279696 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.279930 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:54:52.280413 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.257957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57286)
0929 00:54:52.259408 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.259426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.259553 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.260036 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.260041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.260778 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.261246 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.261254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.262660 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.262666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.264549 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.264556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.265835 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.265840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.266079 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.271537 (+ 5458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.271564 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.272844 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.276810 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.276815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.276820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.277332 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.277719 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.277726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.277731 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.277878 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.278531 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.278535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.278657 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.279698 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.279963 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":138,"thread_start_us":504,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 00:54:52.300858 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.287277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.0.97.61:34477)
0929 00:54:52.287446 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.287451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.287470 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:54:52.287637 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.287642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.287739 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.287842 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.288931 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.289639 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.290950 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.291571 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.291687 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.295196 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.295201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.295209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.295238 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.298151 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.298819 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.298826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.298832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.298965 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.299337 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.299344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.299347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.299634 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.299799 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.300229 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.300447 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:54:52.300912 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.286521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57298)
0929 00:54:52.287248 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.287291 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.287433 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.287886 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.287894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.288559 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.288791 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.288800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.289787 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.289794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.290777 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.290787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.291920 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.291927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.292141 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.292870 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.292887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.295011 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.298290 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.298304 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.298311 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.298689 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.299078 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.299083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.299087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.299231 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.299772 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.299780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.299953 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.300132 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.300450 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":300,"threads_started":1}
I20260929 00:54:52.303181 6523 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34477: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "05a723e6d0384131b6a81a2520b9d979"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34477
}
attrs {
promote: true
}
}
dest_uuid: "bfad066ad17245efbe1c29c65730bb11"
cas_config_opid_index: -1
I20260929 00:54:52.304248 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } }
I20260929 00:54:52.310647 7081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } }
I20260929 00:54:52.311887 7026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:52.315186 7081 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34477
I20260929 00:54:52.315682 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } }
I20260929 00:54:52.315887 6984 master.cc:537] Master@127.0.97.61:34477 shutting down...
I20260929 00:54:52.315959 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:52.317701 7026 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 00:54:52.318753 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.60.keytab.
W20260929 00:54:52.322504 6507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 00:54:52.323235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37443
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:52.333256 6984 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:52.333727 6984 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:52.333920 6984 tablet_replica.cc:330] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: stopping tablet replica
I20260929 00:54:52.341599 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.341078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57310)
0929 00:54:52.341256 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.341378 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 00:54:52.341949 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 -> Peer 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61:34477): Couldn't send request to peer 05a723e6d0384131b6a81a2520b9d979. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 00:54:52.342319 6984 master.cc:552] Master@127.0.97.61:34477 shutdown complete.
I20260929 00:54:52.342410 6984 master_runner.cc:305] Clearing existing system tablet
I20260929 00:54:52.342857 6984 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:52.342945 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:52.346004 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:54:52.347677 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:52.348009 6984 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:52.348278 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:54:52.348359 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "05a723e6d0384131b6a81a2520b9d979"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "380b9167eb55e52abdcea073204a6c30"
server_key_iv: "7b5156c9bc766aac94e9cf65c6254d86"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.353485 6984 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:52.360411 6984 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:54:52.360603 6984 log.cc:1196] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 00:54:52.361078 6984 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Deleting consensus metadata
I20260929 00:54:52.361351 6984 master_runner.cc:315] Copying system tablet from 127.0.97.62:46767
I20260929 00:54:52.363792 6984 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46767
I20260929 00:54:52.382932 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.364643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.0.97.62:46767)
0929 00:54:52.364835 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.364843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.369106 (+ 4263us) server_negotiation.cc:407] Connection header received
0929 00:54:52.369271 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.369278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.369381 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.369510 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.370499 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.371589 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.373438 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.374048 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.374179 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.376327 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.376333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.376338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.376375 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.379960 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.380771 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.380776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.380779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.380931 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.381375 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.381380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.381383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.381733 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.381880 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.382354 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.382575 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:54:52.383023 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.364562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39690)
0929 00:54:52.368912 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.368934 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.369130 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.369538 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.369543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.370039 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.370321 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.370333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.371755 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.371762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.373280 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.373295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.374232 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.374240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.374400 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.375028 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.375049 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.376113 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.380125 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.380130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.380135 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.380642 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.381045 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.381049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.381054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.381240 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.381875 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.381881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.382051 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.382228 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.382526 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4196,"thread_start_us":4107,"threads_started":1}
I20260929 00:54:52.384158 6543 tablet_copy_service.cc:140] P bfad066ad17245efbe1c29c65730bb11: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 05a723e6d0384131b6a81a2520b9d979 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39690)
I20260929 00:54:52.384491 6543 tablet_copy_service.cc:161] P bfad066ad17245efbe1c29c65730bb11: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 05a723e6d0384131b6a81a2520b9d979 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39690: session id = 05a723e6d0384131b6a81a2520b9d979-00000000000000000000000000000000
I20260929 00:54:52.389725 6543 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:54:52.392046 6984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:54:52.398805 6984 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: tablet copy: Starting download of 0 data blocks...
I20260929 00:54:52.399098 6984 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: tablet copy: Starting download of 1 WAL segments...
I20260929 00:54:52.401810 6984 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:54:52.406136 6543 tablet_copy_service.cc:342] P bfad066ad17245efbe1c29c65730bb11: Request end of tablet copy session 05a723e6d0384131b6a81a2520b9d979-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39690
I20260929 00:54:52.406374 6543 tablet_copy_service.cc:433] P bfad066ad17245efbe1c29c65730bb11: ending tablet copy session 05a723e6d0384131b6a81a2520b9d979-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 05a723e6d0384131b6a81a2520b9d979
I20260929 00:54:52.416489 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:52.419235 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:52.428856 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:52.429915 6984 server_base.cc:1034] running on GCE node
W20260929 00:54:52.430503 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:52.430939 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:52.431562 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:52.432763 6984 hybrid_clock.cc:648] HybridClock initialized: now 1790643292432705 us; error 34 us; skew 500 ppm
I20260929 00:54:52.432981 6984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:52.435025 6984 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 00:54:52.436345 6984 webserver.cc:460] Webserver started at http://127.0.97.61:35163/ using document root <none> and password file <none>
I20260929 00:54:52.436758 6984 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:52.436851 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:52.439121 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 00:54:52.440814 7104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:52.441196 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 00:54:52.441358 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "05a723e6d0384131b6a81a2520b9d979"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "380b9167eb55e52abdcea073204a6c30"
server_key_iv: "7b5156c9bc766aac94e9cf65c6254d86"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.441527 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:52.458369 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:52.461660 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:52.498006 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34477
I20260929 00:54:52.498183 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34477 every 8 connection(s)
I20260929 00:54:52.498955 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
I20260929 00:54:52.506089 7156 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:54:52.508551 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Bootstrap starting.
I20260929 00:54:52.521198 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:52.521659 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Bootstrap complete.
I20260929 00:54:52.522536 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } }
I20260929 00:54:52.522773 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05a723e6d0384131b6a81a2520b9d979, State: Initialized, Role: LEARNER
I20260929 00:54:52.523002 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } }
I20260929 00:54:52.523805 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } }
I20260929 00:54:52.524101 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:52.524228 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:54:52.535166 7170 catalog_manager.cc:1259] Loaded cluster ID: 92e065524b9d4501af31b5d644ca9324
I20260929 00:54:52.535293 7170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: loading cluster ID for follower catalog manager: success
I20260929 00:54:52.537853 7170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: acquiring CA information for follower catalog manager: success
I20260929 00:54:52.539835 7170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:54:52.677040 6979 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
W20260929 00:54:52.728294 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:52.728667 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:52.728807 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:52.736435 7085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:54:52.736590 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:52.736678 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:52.736761 7085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:54:52.736819 7085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:54:52.747428 7085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:52.748905 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:52.750344 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:52.762373 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:52.762470 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:52.762398 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:52.763376 7085 server_base.cc:1034] running on GCE node
I20260929 00:54:52.764067 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:52.765452 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:52.766642 7085 hybrid_clock.cc:648] HybridClock initialized: now 1790643292766608 us; error 40 us; skew 500 ppm
I20260929 00:54:52.766966 7085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:52.769969 7085 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 00:54:52.771315 7085 webserver.cc:460] Webserver started at http://127.0.97.60:44913/ using document root <none> and password file <none>
I20260929 00:54:52.772454 7085 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:52.772644 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:52.772998 7085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:54:52.775887 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/instance:
uuid: "f0768315e14c4cab9beb8fc2bdb59238"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "0ab8b72e2c659a539264d601cd201ce1"
server_key_iv: "50791314584ebca41846eee9cabfa7d8"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.776712 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal/instance:
uuid: "f0768315e14c4cab9beb8fc2bdb59238"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "0ab8b72e2c659a539264d601cd201ce1"
server_key_iv: "50791314584ebca41846eee9cabfa7d8"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.783174 7085 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 00:54:52.787690 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:52.789196 7085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:52.789390 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "f0768315e14c4cab9beb8fc2bdb59238"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "0ab8b72e2c659a539264d601cd201ce1"
server_key_iv: "50791314584ebca41846eee9cabfa7d8"
server_key_version: "encryptionkey@0"
I20260929 00:54:52.789585 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:52.812036 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:52.816015 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:52.816295 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:52.835235 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37443
I20260929 00:54:52.835239 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37443 every 8 connection(s)
I20260929 00:54:52.836812 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
I20260929 00:54:52.841034 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:52.844223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 7085
I20260929 00:54:52.848632 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:54:52.867782 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.851969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.97.62:46767)
0929 00:54:52.852190 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.852197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.852732 (+ 535us) server_negotiation.cc:407] Connection header received
0929 00:54:52.852926 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.852930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.852988 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.853066 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.854838 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.855864 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.857773 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.858221 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.858354 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.862026 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.862030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.862033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.862089 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.864709 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.865396 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.865407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.865410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.865526 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.866339 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.866344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.866347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.866587 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.866728 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.867228 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.867435 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:54:52.868585 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.851847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39698)
0929 00:54:52.852569 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.852622 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.852809 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.853155 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.853162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.854456 (+ 1294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.854685 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.854693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.855990 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.855994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.857618 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.857628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.858442 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.858448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.858595 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.859193 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.859230 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.861844 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.864843 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.864854 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.864865 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.865274 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.865639 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.865655 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.865658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.866224 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.866704 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.866712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.866887 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.867103 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.867592 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":194,"thread_start_us":328,"threads_started":1}
I20260929 00:54:52.868614 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.846339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.0.97.60:37443)
0929 00:54:52.847052 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.847066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.847122 (+ 56us) server_negotiation.cc:407] Connection header received
0929 00:54:52.849279 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.849312 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.849842 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.850277 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.851724 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.854251 (+ 2527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.855922 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.857070 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.857250 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.861589 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.861597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.861609 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.861637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.864145 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.864889 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.864901 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.864904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.865058 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.865427 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.865433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.865436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.866240 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.866508 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.866995 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.867455 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":500,"spinlock_wait_cycles":19200,"thread_start_us":287,"threads_started":1}
I20260929 00:54:52.870110 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 00:54:52.882789 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.870762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57314)
0929 00:54:52.870931 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.870950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.871041 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.872038 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.872042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.872398 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.872593 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.872601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.873849 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.873852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.874764 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.874774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.875364 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.875368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.875517 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.875985 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.876000 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.877840 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.880666 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.880672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.880677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.881249 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.881646 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.881650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.881652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.881767 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.882185 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.882188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.882289 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.882388 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.882570 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 00:54:52.882879 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.870905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.97.61:34477)
0929 00:54:52.871800 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.871805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.871819 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:54:52.871864 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.871867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.871920 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.871999 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.872715 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.873743 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.874877 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.875257 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.875343 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.878003 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.878008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.878011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.878041 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.880468 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.881375 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.881380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.881384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.881522 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.881857 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.881860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.881862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.882093 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.882205 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.882469 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.882608 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":689,"threads_started":1}
I20260929 00:54:52.883770 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 00:54:52.895188 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.884423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.0.1:54188)
0929 00:54:52.884637 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.884661 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.884730 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.884947 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.884951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.885241 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.885370 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.885376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.886235 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.886238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.887019 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.887028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.888099 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.888103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.888253 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.888849 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.888865 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.890675 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.893249 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.893252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.893255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.893576 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.893919 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.893924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.893927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.894063 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.894489 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.894492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.894640 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.894747 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.894913 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 00:54:52.895269 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.884557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.0.97.60:37443)
0929 00:54:52.884693 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.884696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.884716 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:54:52.884798 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.884801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.884847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.884917 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.885479 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.886138 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.887127 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.887967 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.888076 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.890845 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.890851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.890856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.890891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.893142 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.893671 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.893675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.893677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.893780 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.894158 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.894161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.894163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.894394 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.894496 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.894820 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.895026 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":31}
I20260929 00:54:52.899744 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Bootstrap starting.
I20260929 00:54:52.902918 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:52.903961 7237 log.cc:824] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:52.906407 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: No bootstrap required, opened a new log
I20260929 00:54:52.913826 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } }
I20260929 00:54:52.914131 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:52.914252 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0768315e14c4cab9beb8fc2bdb59238, State: Initialized, Role: FOLLOWER
I20260929 00:54:52.914884 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } }
I20260929 00:54:52.915860 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } } }
I20260929 00:54:52.916203 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:52.916990 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:52.920548 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.914586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.61:34477)
0929 00:54:52.914903 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.914911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.914962 (+ 51us) server_negotiation.cc:407] Connection header received
0929 00:54:52.915145 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.915150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.915263 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.915407 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:52.915907 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.916966 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.918631 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.919616 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.919751 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.919961 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.920161 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.920287 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260929 00:54:52.920604 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.914383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57322)
0929 00:54:52.914886 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.914902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.915009 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.915451 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.915458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.915493 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:52.915749 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.915762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.917129 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.917135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.918487 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.918498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.919785 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.919791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.919966 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.920030 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.920135 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.920267 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260929 00:54:52.921693 7131 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 00:54:52.922422 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfad066ad17245efbe1c29c65730bb11. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } }
I20260929 00:54:52.922665 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:54:52.923341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6584
W20260929 00:54:52.927939 7257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:54:52.928082 7257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:54:52.930656 7085 master_runner.cc:428] Detected that this master f0768315e14c4cab9beb8fc2bdb59238 is joining an existing cluster
I20260929 00:54:52.930779 7085 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:37443
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 00:54:52.951915 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.924955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.97.1:43427)
0929 00:54:52.929004 (+ 4049us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.929010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.929032 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:54:52.929113 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.929121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.929384 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.929633 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.931244 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.933056 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.935003 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.935864 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.935987 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.942053 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.942067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.942083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.942121 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.947024 (+ 4903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.948058 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.948072 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.948078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.948282 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.948826 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.948834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.948839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.949372 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.949714 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.951149 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.951464 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3876,"thread_start_us":3729,"threads_started":1}
I20260929 00:54:52.953624 6707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:52.963321 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.936342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.0.97.61:34477)
0929 00:54:52.936569 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.936576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.937309 (+ 733us) server_negotiation.cc:407] Connection header received
0929 00:54:52.937556 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.937561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.937637 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.937701 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.939178 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.940249 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.942323 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.943466 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.943569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.953739 (+ 10170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.953748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.953756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.953800 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.959408 (+ 5608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.960574 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.960583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.960590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.960777 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.961331 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.961339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.961345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.961739 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.961903 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.962606 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.962905 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 00:54:52.963971 7246 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:52.964367 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.936131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:57338)
0929 00:54:52.937217 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.937236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.937409 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.937908 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.937914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.938596 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.938988 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.939001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.940416 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.940423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.942144 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.942159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.943855 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.943863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.944055 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.952220 (+ 8165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.952243 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.953507 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.959616 (+ 6109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.959624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.959631 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.960375 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.960949 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.960956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.960963 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.961185 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.961995 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.962002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.962278 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.963563 (+ 1285us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.963841 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":735,"threads_started":1}
I20260929 00:54:52.964287 7246 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } }
I20260929 00:54:52.966073 7246 leader_election.cc:288] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61:34477)
I20260929 00:54:52.967505 6533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0768315e14c4cab9beb8fc2bdb59238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad066ad17245efbe1c29c65730bb11" is_pre_election: true
I20260929 00:54:52.968287 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.936950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39714)
0929 00:54:52.951582 (+ 14632us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.951601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.951757 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.952210 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.952215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.953015 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.953389 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.953401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.954920 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.954925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.957081 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.957095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.958299 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.958306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.958475 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.959100 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.959122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.960275 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.964137 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.964142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.964146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.964883 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.965322 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.965326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.965329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.965509 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.966232 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.966237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.966438 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.967539 (+ 1101us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.967858 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14479,"thread_start_us":14261,"threads_started":1}
I20260929 00:54:52.970731 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0768315e14c4cab9beb8fc2bdb59238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a723e6d0384131b6a81a2520b9d979" is_pre_election: true
I20260929 00:54:52.970968 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.935978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.0.97.62:46767)
0929 00:54:52.936170 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.936177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.951719 (+ 15542us) server_negotiation.cc:407] Connection header received
0929 00:54:52.951871 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.951878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.951954 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.952044 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.953565 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.954698 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.957258 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.957906 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.958008 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.960492 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.960518 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.960523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.960564 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.963970 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.965017 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.965022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.965025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.965166 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.965639 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.965644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.965647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.966009 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.966157 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.966766 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.966990 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 00:54:52.971496 7186 leader_election.cc:302] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0768315e14c4cab9beb8fc2bdb59238; no voters: 05a723e6d0384131b6a81a2520b9d979, bfad066ad17245efbe1c29c65730bb11
I20260929 00:54:52.972208 7246 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 29 00:54:52 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643290, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 00:54:52.991861 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.944825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.97.60:37443)
0929 00:54:52.944987 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.944994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.951879 (+ 6885us) server_negotiation.cc:407] Connection header received
0929 00:54:52.963190 (+ 11311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.963196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.963258 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.963323 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.968559 (+ 5236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.970056 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.976568 (+ 6512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.977503 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.977635 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.981760 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.981767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.981772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.981805 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.988158 (+ 6353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.989355 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.989360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.989364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.989520 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.989954 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.989959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.989962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.990342 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.990458 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.991004 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.991259 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:54:52.992726 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.945158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.0.1:54194)
0929 00:54:52.951792 (+ 6634us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.951810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.962923 (+ 11113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.964913 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.964918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.965497 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.965847 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.965857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.974168 (+ 8311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.974176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.976317 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.976335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.979109 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.979118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.979325 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.980341 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.980365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.981588 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.988669 (+ 7081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.988674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.988679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.989221 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.989660 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.989665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.989669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.989833 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.990529 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.990533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.990718 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.991213 (+ 495us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.992180 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6522,"thread_start_us":6732,"threads_started":1}
I20260929 00:54:52.997237 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.976647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.0.1:54208)
0929 00:54:52.976820 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:52.976841 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:52.976938 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:52.978464 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:52.978470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:52.978953 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:52.979132 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.979145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.981583 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.981589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.983017 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:52.983031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.984079 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.984086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:52.984271 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.984871 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:52.984895 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:52.989055 (+ 4160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:52.994644 (+ 5589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.994648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.994652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.995140 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.995534 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:52.995539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:52.995543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:52.995683 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:52.996261 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:52.996266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:52.996425 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 00:54:52.996619 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 00:54:52.996881 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 00:54:52.997296 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:52.976988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.0.97.60:37443)
0929 00:54:52.977598 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:52.977605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:52.977627 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:54:52.977719 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:52.977724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:52.977817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:52.979283 (+ 1466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:52.980063 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.981418 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.983210 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:52.983893 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:52.984563 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:52.989343 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:52.989352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:52.989358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:52.989441 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:52.994522 (+ 5081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.995271 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.995277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.995280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.995418 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:52.995818 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:52.995823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:52.995827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:52.996132 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:52.996262 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:52.996712 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 00:54:52.996907 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"spinlock_wait_cycles":41728,"thread_start_us":336,"threads_started":1}
I20260929 00:54:52.998838 6523 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:37443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 37443
}
attrs {
promote: true
}
}
dest_uuid: "bfad066ad17245efbe1c29c65730bb11"
cas_config_opid_index: 5
I20260929 00:54:52.999850 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } }
I20260929 00:54:53.002128 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:54:53.002156 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.003048 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 00:54:53.017897 7244 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:37443
I20260929 00:54:53.018465 7269 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.018692 7212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:53.018944 7269 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.019340 7085 master.cc:537] Master@127.0.97.60:37443 shutting down...
I20260929 00:54:53.020138 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.024307 7212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 00:54:53.024623 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: true } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.024886 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:54:53.025161 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:53.026486 7244 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: attempting to promote NON_VOTER 05a723e6d0384131b6a81a2520b9d979 to VOTER
I20260929 00:54:53.029362 7244 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } }
I20260929 00:54:53.038197 7085 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:53.038570 7085 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:53.038753 7085 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: stopping tablet replica
I20260929 00:54:53.040947 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 LEARNER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 00:54:53.041708 7271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 00:54:53.044291 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.044883 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.048259 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.049122 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.048782 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.049336 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
W20260929 00:54:53.050555 6510 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 -> Peer f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60:37443): Couldn't send request to peer f0768315e14c4cab9beb8fc2bdb59238. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 00:54:53.052661 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.052886 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:53.055101 7085 master.cc:552] Master@127.0.97.60:37443 shutdown complete.
I20260929 00:54:53.055189 7085 master_runner.cc:305] Clearing existing system tablet
I20260929 00:54:53.055552 7085 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:53.055647 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:53.058674 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:54:53.060228 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:53.060575 7085 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:53.060966 7085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:54:53.061102 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "f0768315e14c4cab9beb8fc2bdb59238"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "0ab8b72e2c659a539264d601cd201ce1"
server_key_iv: "50791314584ebca41846eee9cabfa7d8"
server_key_version: "encryptionkey@0"
I20260929 00:54:53.065966 7085 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:53.072957 7085 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 00:54:53.073201 7085 log.cc:1196] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 00:54:53.073745 7085 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Deleting consensus metadata
I20260929 00:54:53.074066 7085 master_runner.cc:315] Copying system tablet from 127.0.97.62:46767
I20260929 00:54:53.076596 7085 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46767
I20260929 00:54:53.093618 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.077811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.0.97.62:46767)
0929 00:54:53.078018 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.078024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.078516 (+ 492us) server_negotiation.cc:407] Connection header received
0929 00:54:53.078948 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.078954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.079035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.079105 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:53.080635 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.081780 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.083676 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.084282 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.084409 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.086819 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:53.086826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:53.086831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:53.086879 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:53.090549 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.091339 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.091345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.091349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.091564 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.091999 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.092004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.092008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.092383 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:53.092518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.093090 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.093294 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 00:54:53.094926 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.077747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:39728)
0929 00:54:53.078405 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:53.078425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:53.078744 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:53.079308 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:53.079313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:53.080204 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:53.080450 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.080462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.081978 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.081984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.083509 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.083525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.084472 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.084480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.084733 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.085461 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:53.085503 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:53.086615 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:53.090706 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.090711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.090716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.091205 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.091713 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.091718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.091722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.091881 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.092660 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:53.092665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:53.092837 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 00:54:53.094151 (+ 1314us) client_negotiation.cc:240] Negotiation successful
0929 00:54:53.094505 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":385,"threads_started":1}
I20260929 00:54:53.095719 6543 tablet_copy_service.cc:140] P bfad066ad17245efbe1c29c65730bb11: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0768315e14c4cab9beb8fc2bdb59238 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39728)
I20260929 00:54:53.095880 6543 tablet_copy_service.cc:161] P bfad066ad17245efbe1c29c65730bb11: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0768315e14c4cab9beb8fc2bdb59238 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39728: session id = f0768315e14c4cab9beb8fc2bdb59238-00000000000000000000000000000000
I20260929 00:54:53.099324 6543 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 00:54:53.101248 7085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 00:54:53.107867 7085 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: tablet copy: Starting download of 0 data blocks...
I20260929 00:54:53.108282 7085 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: tablet copy: Starting download of 1 WAL segments...
I20260929 00:54:53.111131 7085 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 00:54:53.115532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6584
I20260929 00:54:53.117322 6543 tablet_copy_service.cc:342] P bfad066ad17245efbe1c29c65730bb11: Request end of tablet copy session f0768315e14c4cab9beb8fc2bdb59238-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39728
I20260929 00:54:53.117488 6543 tablet_copy_service.cc:433] P bfad066ad17245efbe1c29c65730bb11: ending tablet copy session f0768315e14c4cab9beb8fc2bdb59238-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0768315e14c4cab9beb8fc2bdb59238
I20260929 00:54:53.130949 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:53.140343 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:53.140664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43427
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46767
--webserver_port=46069
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:54:53.148239 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:53.148881 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:53.148957 7085 server_base.cc:1034] running on GCE node
I20260929 00:54:53.149478 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:53.150159 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:53.151439 7085 hybrid_clock.cc:648] HybridClock initialized: now 1790643293151398 us; error 34 us; skew 500 ppm
I20260929 00:54:53.151684 7085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:53.153966 7085 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 00:54:53.155331 7085 webserver.cc:460] Webserver started at http://127.0.97.60:41851/ using document root <none> and password file <none>
I20260929 00:54:53.155666 7085 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:53.155802 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:53.161229 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:54:53.163195 7294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:53.163630 7085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 00:54:53.163757 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "f0768315e14c4cab9beb8fc2bdb59238"
format_stamp: "Formatted at 2026-09-29 00:54:52 on dist-test-slave-pzsx"
server_key: "0ab8b72e2c659a539264d601cd201ce1"
server_key_iv: "50791314584ebca41846eee9cabfa7d8"
server_key_version: "encryptionkey@0"
I20260929 00:54:53.163942 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:53.203856 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:53.209095 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:53.245409 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37443
I20260929 00:54:53.245471 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37443 every 8 connection(s)
I20260929 00:54:53.246467 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
I20260929 00:54:53.251773 7346 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:54:53.255466 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Bootstrap starting.
I20260929 00:54:53.273718 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:54:53.274287 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: Bootstrap complete.
I20260929 00:54:53.275354 7346 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } }
I20260929 00:54:53.275669 7346 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0768315e14c4cab9beb8fc2bdb59238, State: Initialized, Role: LEARNER
I20260929 00:54:53.275914 7346 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } }
I20260929 00:54:53.277017 7346 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:54:53.276770 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.277268 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: LEARNER
I20260929 00:54:53.287168 7360 catalog_manager.cc:1259] Loaded cluster ID: 92e065524b9d4501af31b5d644ca9324
I20260929 00:54:53.287318 7360 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: loading cluster ID for follower catalog manager: success
I20260929 00:54:53.291107 7360 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: acquiring CA information for follower catalog manager: success
I20260929 00:54:53.293398 7360 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:54:53.478776 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.471225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.0.1:54214)
0929 00:54:53.471407 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:53.471428 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:53.471566 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:53.472389 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:53.472396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:53.472423 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:53.472881 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.472892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.474419 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.474425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.476315 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.476350 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.477947 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.477956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.478163 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.478200 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:54:53.478311 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:54:53.478459 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:54:53.478801 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.471412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.97.60:37443)
0929 00:54:53.472072 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.472076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.472091 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:54:53.472146 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.472150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.472222 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.472355 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:53.473044 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.474256 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.476542 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.477747 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.477912 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.478180 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.478341 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.478468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":419,"threads_started":1}
I20260929 00:54:53.480113 7321 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 00:54:53.480820 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfad066ad17245efbe1c29c65730bb11. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.481060 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: LEARNER
W20260929 00:54:53.511890 7288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:53.512140 7288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:53.512296 7288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:53.519562 7288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:53.519660 7288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:53.519827 7288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:54:53.530038 7288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46069
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:53.531440 7288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:53.532780 7288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:53.546097 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:53.546227 7369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:53.546460 7366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:53.547151 7288 server_base.cc:1034] running on GCE node
I20260929 00:54:53.547731 7288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:53.548679 7288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:53.549897 7288 hybrid_clock.cc:648] HybridClock initialized: now 1790643293549809 us; error 72 us; skew 500 ppm
I20260929 00:54:53.550225 7288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:53.553314 7288 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 00:54:53.554555 7288 webserver.cc:460] Webserver started at http://127.0.97.1:46069/ using document root <none> and password file <none>
I20260929 00:54:53.555172 7288 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:53.555282 7288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:53.560770 7288 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:54:53.563277 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:53.564661 7288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:53.564795 7288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "26f9db9faab8437e8608c109ee3e9837"
format_stamp: "Formatted at 2026-09-29 00:54:50 on dist-test-slave-pzsx"
server_key: "27ad0d37738312eea344bc04cfab82a3"
server_key_iv: "f58cd4d1028d0a144a688c0b76a324c5"
server_key_version: "encryptionkey@0"
I20260929 00:54:53.565726 7288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:53.579371 7288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:53.583300 7288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:53.583518 7288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:53.584800 7288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:53.586045 7288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:53.586127 7288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:53.586201 7288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:53.586256 7288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:53.620733 7288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43427
I20260929 00:54:53.620777 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43427 every 8 connection(s)
I20260929 00:54:53.622164 7288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:54:53.627106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 7288
I20260929 00:54:53.627487 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6718
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 00:54:53 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 00:54:53.661998 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.629134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.0.97.2:45467)
0929 00:54:53.635057 (+ 5923us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.635069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.639149 (+ 4080us) server_negotiation.cc:407] Connection header received
0929 00:54:53.639229 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.639237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.639512 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.639675 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:53.642124 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.643784 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.645780 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.646630 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.646723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.653037 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:53.653050 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:53.653066 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:53.653123 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:53.657833 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.658729 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.658739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.658743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.658899 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.659351 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.659358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.659363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.659814 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:53.660101 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.661295 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.661561 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5707,"thread_start_us":5531,"threads_started":1}
I20260929 00:54:53.663439 6841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:53.664297 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.625059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33617 (local address 127.0.97.62:46767)
0929 00:54:53.637827 (+ 12768us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.637835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.641234 (+ 3399us) server_negotiation.cc:407] Connection header received
0929 00:54:53.643181 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.643190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.643318 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.643462 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:53.646221 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.647783 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.650073 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.650697 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.650785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.656112 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:53.656119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:53.656124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:53.656164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:53.660621 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.661614 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.661621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.661628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.661784 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.662331 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.662338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.662344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.662677 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:53.662819 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.663760 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.663976 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12625,"thread_start_us":12541,"threads_started":1}
I20260929 00:54:53.665993 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.625059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33407 (local address 127.0.97.61:34477)
0929 00:54:53.638248 (+ 13189us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.638257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.642831 (+ 4574us) server_negotiation.cc:407] Connection header received
0929 00:54:53.643371 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.643377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.643462 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.643546 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:53.646206 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.647265 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.651928 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.652629 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.652767 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.658266 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:53.658274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:53.658278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:53.658321 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:53.661835 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.662623 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.662628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.662632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.662772 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.663151 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.663156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.663159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.663508 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:53.663650 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.664995 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.665202 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13025,"thread_start_us":13186,"threads_started":1}
I20260929 00:54:53.666507 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.625745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:33617)
0929 00:54:53.639445 (+ 13700us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:53.639499 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:53.642870 (+ 3371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:53.643996 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:53.644004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:53.645049 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:53.646003 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.646027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.647961 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.647966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.649904 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.649917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.651099 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.651106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.651405 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.652127 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:53.652158 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:53.655844 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:53.660824 (+ 4980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.660832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.660853 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.661463 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.661933 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.661938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.661942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.662185 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.662906 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:53.662913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:53.663339 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 00:54:53.664468 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0929 00:54:53.664767 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13361,"mutex_wait_us":34,"thread_start_us":13137,"threads_started":1}
I20260929 00:54:53.666561 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.625745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.97.1:33407)
0929 00:54:53.642700 (+ 16955us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:53.642719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:53.643221 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:53.643991 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:53.644003 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:53.645498 (+ 1495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:53.646015 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.646028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.647521 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.647526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.651799 (+ 4273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.651813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.653048 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.653054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.653318 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.654221 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:53.654239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:53.658000 (+ 3761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:53.661959 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.661963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.661966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.662487 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.662891 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.662896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.662899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.663027 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.663755 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:53.663759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:53.663916 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 00:54:53.664061 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 00:54:53.664581 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16788,"mutex_wait_us":2216,"spinlock_wait_cycles":24192,"thread_start_us":18658,"threads_started":1}
I20260929 00:54:53.668115 7490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:53.668376 7491 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34477
I20260929 00:54:53.668519 7490 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:53.668702 7491 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:53.669535 7490 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
I20260929 00:54:53.669718 7491 heartbeater.cc:511] Master 127.0.97.61:34477 requested a full tablet report, sending...
I20260929 00:54:53.670135 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.625421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41539 (local address 127.0.97.60:37443)
0929 00:54:53.625930 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:53.625938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:53.645040 (+ 19102us) server_negotiation.cc:407] Connection header received
0929 00:54:53.645146 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:53.645154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:53.645238 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:53.645336 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:53.647014 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.648081 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.652003 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.652661 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.652767 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.657791 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:53.657797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:53.657802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:53.657838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:53.661104 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.661937 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.661944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.661948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.662087 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:53.663016 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:53.663020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:53.663023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:53.663359 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:53.663479 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:53.664224 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 00:54:53.667855 (+ 3631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260929 00:54:53.671501 6523 ts_manager.cc:194] Re-registered known tserver with Master: 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427)
I20260929 00:54:53.671757 7121 ts_manager.cc:194] Registered new tserver with Master: 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427)
I20260929 00:54:53.673009 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33617
I20260929 00:54:53.673318 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:53.626421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.97.1:41539)
0929 00:54:53.644909 (+ 18488us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:53.644928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:53.645039 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:53.645488 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:53.645493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:53.646126 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:53.646849 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.646860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.649409 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.649416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.651048 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:53.651066 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:53.653028 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:53.653035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:53.653316 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:53.654147 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:53.654177 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:53.657532 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:53.661255 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.661260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.661263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.661804 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.662492 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:53.662496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:53.662499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:53.662889 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:53.663703 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:53.663711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:53.663896 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 00:54:53.670707 (+ 6811us) client_negotiation.cc:240] Negotiation successful
0929 00:54:53.672990 (+ 2283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18376,"mutex_wait_us":88,"spinlock_wait_cycles":20736,"thread_start_us":16007,"threads_started":1}
I20260929 00:54:53.674414 7489 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37443
I20260929 00:54:53.674573 7489 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:53.675005 7489 heartbeater.cc:511] Master 127.0.97.60:37443 requested a full tablet report, sending...
I20260929 00:54:53.676589 7311 ts_manager.cc:194] Registered new tserver with Master: 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427)
I20260929 00:54:53.758296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6718
I20260929 00:54:53.775954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45467
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46767
--webserver_port=39903
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:53.967334 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: true } } }
I20260929 00:54:53.967746 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.969167 7244 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: attempting to promote NON_VOTER f0768315e14c4cab9beb8fc2bdb59238 to VOTER
I20260929 00:54:53.969779 7244 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:53.971516 7321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 LEARNER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 00:54:53.971665 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Refusing update from remote peer bfad066ad17245efbe1c29c65730bb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 00:54:53.972095 7503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 00:54:53.972585 7502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 00:54:53.975168 7503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.977244 7321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.980058 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.980340 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.980279 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.980569 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:53.980628 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.980911 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.981956 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.984933 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.985134 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:53.985906 7507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfad066ad17245efbe1c29c65730bb11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:53.986279 7507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 00:54:54.105011 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:54.105302 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:54.105458 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:54.112452 7501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:54.112661 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:54.112834 7501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:54:54.122447 7501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39903
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:54.123791 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:54.125294 7501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:54.138350 7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:54.138268 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:54.138322 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:54.139070 7501 server_base.cc:1034] running on GCE node
I20260929 00:54:54.139660 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:54.140661 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:54.141870 7501 hybrid_clock.cc:648] HybridClock initialized: now 1790643294141828 us; error 54 us; skew 500 ppm
I20260929 00:54:54.142189 7501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:54.145371 7501 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 00:54:54.146685 7501 webserver.cc:460] Webserver started at http://127.0.97.2:39903/ using document root <none> and password file <none>
I20260929 00:54:54.147342 7501 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:54.147495 7501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:54.152834 7501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:54:54.155428 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.156874 7501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:54:54.157039 7501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "87284e1bb781498db54719968ba55cd8"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "0c3292a992cb1926d80208b23e82dd6a"
server_key_iv: "a33ad42d6da09c98364ef739b75c84d7"
server_key_version: "encryptionkey@0"
I20260929 00:54:54.157968 7501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:54.173841 7501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:54.178037 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:54.178357 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:54.179267 7501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:54.180660 7501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:54.180773 7501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.180899 7501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:54.180986 7501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.212996 7501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45467
I20260929 00:54:54.213095 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45467 every 8 connection(s)
I20260929 00:54:54.214450 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:54:54.216110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 7501
I20260929 00:54:54.216404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6851
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 00:54:54.253929 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.229190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.97.2:49641)
0929 00:54:54.230726 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.230754 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.230956 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.231946 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.231959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.233289 (+ 1330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.234485 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.234498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.235960 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.235965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.238506 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.238520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.240279 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.240288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.240527 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.241326 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.241352 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.245501 (+ 4149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.249413 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.249423 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.249441 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.250049 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.250459 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.250464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.250468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.250668 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.251294 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.251302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.251695 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.251988 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.252489 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"mutex_wait_us":348,"thread_start_us":1221,"threads_started":1}
I20260929 00:54:54.254971 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.229674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49641 (local address 127.0.97.61:34477)
0929 00:54:54.231539 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.231544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.231558 (+ 14us) server_negotiation.cc:407] Connection header received
0929 00:54:54.231605 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.231608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.231671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.231803 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.234742 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.235782 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.238686 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.239367 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.239500 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.245755 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.245762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.245767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.245810 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.249251 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.250217 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.250222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.250226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.250350 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.250785 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.250790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.250794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.251105 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.251233 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.254417 (+ 3184us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.254637 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1667,"threads_started":1}
I20260929 00:54:54.255888 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.217991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.0.97.3:35065)
0929 00:54:54.229190 (+ 11199us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.229199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.230225 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 00:54:54.230296 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.230304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.230564 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.230717 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.233563 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.235192 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.237035 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.237853 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.237971 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.246786 (+ 8815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.246810 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.247100 (+ 290us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.247153 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.251855 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.252786 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.252795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.252800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.252964 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.253405 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.253411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.253415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.253809 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.254107 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.255238 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.255499 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11009,"thread_start_us":11638,"threads_started":1}
I20260929 00:54:54.256407 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.217560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47057 (local address 127.0.97.60:37443)
0929 00:54:54.229546 (+ 11986us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.229553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.229574 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:54:54.232144 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.232153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.232249 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.232338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.234639 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.235936 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.238723 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.239362 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.239455 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.246018 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.246025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.246030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.246078 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.250552 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.251379 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.251385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.251389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.251545 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.251997 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.252001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.252004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.252385 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.252582 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.253975 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.254885 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11858,"thread_start_us":12500,"threads_started":1}
I20260929 00:54:54.256739 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.217983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.97.2:47057)
0929 00:54:54.227707 (+ 9724us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.227756 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.230470 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.232588 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.232592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.233289 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.234429 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.234461 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.236099 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.236123 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.238563 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.238576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.239826 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.239833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.240203 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.241132 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.241156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.245791 (+ 4635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.250736 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.250741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.250744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.251200 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.251682 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.251688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.251692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.251870 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.252646 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.252653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.252848 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.252997 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.253304 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9446,"mutex_wait_us":702,"spinlock_wait_cycles":34944,"thread_start_us":10107,"threads_started":1}
I20260929 00:54:54.256925 7637 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34477
I20260929 00:54:54.257251 7637 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.258002 7637 heartbeater.cc:511] Master 127.0.97.61:34477 requested a full tablet report, sending...
I20260929 00:54:54.259042 6974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:54.259277 7635 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37443
I20260929 00:54:54.259403 7635 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.259711 7121 ts_manager.cc:194] Registered new tserver with Master: 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467)
I20260929 00:54:54.259826 7635 heartbeater.cc:511] Master 127.0.97.60:37443 requested a full tablet report, sending...
I20260929 00:54:54.261073 7311 ts_manager.cc:194] Registered new tserver with Master: 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467)
I20260929 00:54:54.266182 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.217277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57403 (local address 127.0.97.62:46767)
0929 00:54:54.218424 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.218434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.228196 (+ 9762us) server_negotiation.cc:407] Connection header received
0929 00:54:54.230944 (+ 2748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.230950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.231037 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.231116 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.234710 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.235837 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.238723 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.239363 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.239455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.259903 (+ 20448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.259913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.259919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.259983 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.263492 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.264162 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.264168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.264171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.264307 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.264798 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.264803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.264806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.265050 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.265204 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.265581 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.265776 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":876,"threads_started":1}
I20260929 00:54:54.266214 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.217983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:57403)
0929 00:54:54.228045 (+ 10062us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.228067 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.230747 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.231959 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.231963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.233293 (+ 1330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.234429 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.234461 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.236060 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.236065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.238565 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.238576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.239826 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.239833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.240208 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.241107 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.241145 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.253175 (+ 12030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.263644 (+ 10469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.263649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.263653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.264002 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.264425 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.264430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.264434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.264675 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.265180 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.265184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.265337 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.265486 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.265776 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9889,"mutex_wait_us":963,"spinlock_wait_cycles":27904,"thread_start_us":9871,"threads_started":1}
I20260929 00:54:54.267004 7636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:54.267145 7636 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.267644 7636 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
I20260929 00:54:54.268585 6523 ts_manager.cc:194] Re-registered known tserver with Master: 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467)
I20260929 00:54:54.269757 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57403
I20260929 00:54:54.348529 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6851
I20260929 00:54:54.364646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35065
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46767
--webserver_port=40577
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:54:54.675745 7490 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
W20260929 00:54:54.689677 7648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:54:54.689981 7648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:54:54.690166 7648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:54:54.697325 7648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:54:54.697475 7648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:54:54.697647 7648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:54:54.707222 7648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40577
--tserver_master_addrs=127.0.97.62:46767,127.0.97.61:34477,127.0.97.60:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:54:54.708602 7648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:54:54.709932 7648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:54:54.722709 7654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:54.722671 7655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:54:54.722743 7657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:54:54.723714 7648 server_base.cc:1034] running on GCE node
I20260929 00:54:54.724436 7648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:54:54.725426 7648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:54:54.726622 7648 hybrid_clock.cc:648] HybridClock initialized: now 1790643294726583 us; error 34 us; skew 500 ppm
I20260929 00:54:54.726944 7648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:54:54.730214 7648 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 00:54:54.731604 7648 webserver.cc:460] Webserver started at http://127.0.97.3:40577/ using document root <none> and password file <none>
I20260929 00:54:54.732230 7648 fs_manager.cc:359] Metadata directory not provided
I20260929 00:54:54.732339 7648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:54:54.737869 7648 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 00:54:54.740399 7664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.741716 7648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 00:54:54.741891 7648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "e4a3178fdc8c49b2a208caad63034b98"
format_stamp: "Formatted at 2026-09-29 00:54:51 on dist-test-slave-pzsx"
server_key: "a9fd4b41ffe4afeb7be840e63d62fb19"
server_key_iv: "86c7fe5049029de93dfb643e2fc65059"
server_key_version: "encryptionkey@0"
I20260929 00:54:54.742894 7648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:54:54.756865 7648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:54:54.760634 7648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:54:54.761018 7648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:54:54.762017 7648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:54:54.763255 7648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:54:54.763336 7648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.763410 7648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:54:54.763466 7648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:54:54.797292 7648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35065
I20260929 00:54:54.797294 7776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35065 every 8 connection(s)
I20260929 00:54:54.799286 7648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:54:54.804772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 7648
I20260929 00:54:54.838778 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.810908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.0.97.61:34477)
0929 00:54:54.811423 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.811431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.811453 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:54:54.811521 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.811527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.811607 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.811731 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.814435 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.815692 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.820113 (+ 4421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.820919 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.821055 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.827795 (+ 6740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.827824 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.827833 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.827967 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.832939 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.834069 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.834078 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.834084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.834384 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.834959 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.834980 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.834985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.835498 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.835914 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.837782 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.838125 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":379,"threads_started":1}
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 00:54:54 dist-test-slave-pzsx krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 00:54:54.865541 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.821505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.0.97.60:37443)
0929 00:54:54.838849 (+ 17344us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.840299 (+ 1450us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.840330 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:54:54.840401 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.840409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.840518 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.840641 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.845079 (+ 4438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.846127 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.848025 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.848761 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.848882 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.852587 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.852592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.852596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.852627 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.857071 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.859076 (+ 2005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.859082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.859086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.859255 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.861073 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.861078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.861082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.861443 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.864487 (+ 3044us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.864662 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.864972 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":17189,"thread_start_us":17055,"threads_started":1}
I20260929 00:54:54.868149 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.811828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.97.62:46767)
0929 00:54:54.845786 (+ 33958us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.845791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.845810 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:54:54.845849 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.845853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.845933 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.846036 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.848125 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.849185 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.852575 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.853168 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.853298 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.857684 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.857690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.857694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.857732 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.861727 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.863905 (+ 2178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.863910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.863914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.864064 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.865168 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.865172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.865175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.865489 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.865694 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.866211 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.867796 (+ 1585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33856,"thread_start_us":34312,"threads_started":1}
I20260929 00:54:54.869575 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.803344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51625 (local address 127.0.97.62:46767)
0929 00:54:54.843971 (+ 40627us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.843977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.843996 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:54:54.844053 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.844060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.844157 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.844599 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.846401 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.847566 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.851347 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.852003 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.852123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.858795 (+ 6672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.858801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.858811 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.858849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.862730 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.864619 (+ 1889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.864624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.864627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.864751 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.866258 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.866264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.866267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.867074 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.867201 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.868315 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.869137 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40508,"thread_start_us":40427,"threads_started":1}
I20260929 00:54:54.871233 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.802794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:51625)
0929 00:54:54.803646 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.803697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.806565 (+ 2868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.844516 (+ 37951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.844523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.845423 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.846178 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.846209 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.848606 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.848609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.851190 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.851206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.853293 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.853302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.853587 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.854221 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.854257 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.858200 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.863493 (+ 5293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.863504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.863523 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.864227 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.865184 (+ 957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.865188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.865191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.866143 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.867266 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.867280 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.867895 (+ 615us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.868454 (+ 559us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.868922 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":80,"thread_start_us":2557,"threads_started":1}
I20260929 00:54:54.871598 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.802247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44287 (local address 127.0.97.60:37443)
0929 00:54:54.802726 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.802733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.841790 (+ 39057us) server_negotiation.cc:407] Connection header received
0929 00:54:54.841967 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.841975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.842082 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.842211 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.847345 (+ 5134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.848407 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.851616 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.852323 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.852456 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.862024 (+ 9568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.862030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.862035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.862079 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.865629 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.867795 (+ 2166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.867802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.867805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.867933 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.869927 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.869932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.869936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.870326 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.870456 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.870972 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.871199 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260929 00:54:54.872516 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.803640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.97.3:44287)
0929 00:54:54.841659 (+ 38019us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.841682 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.841830 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.843557 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.843578 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.846859 (+ 3281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.847156 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.847170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.849420 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.849425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.851464 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.851477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.854896 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.854908 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.855138 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.857337 (+ 2199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.857363 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.861766 (+ 4403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.865759 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.865762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.865765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.867653 (+ 1888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.868030 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.868035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.868040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.869781 (+ 1741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.870542 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.870547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.870700 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.871795 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.872060 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37835,"mutex_wait_us":2,"spinlock_wait_cycles":108672,"thread_start_us":39401,"threads_started":1}
I20260929 00:54:54.873019 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.809479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.97.3:39627)
0929 00:54:54.843394 (+ 33915us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.843415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.843556 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.844334 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.844339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.852051 (+ 7712us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:54:54.852332 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.852344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.853865 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.853870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.855481 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.855497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.856238 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.856244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.856449 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.857044 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:54:54.857066 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:54:54.861493 (+ 4427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:54:54.865570 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.865575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.865578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.867603 (+ 2025us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.868054 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:54:54.868058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:54:54.868061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:54:54.868205 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:54:54.870673 (+ 2468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:54:54.870677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:54:54.870833 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.872523 (+ 1690us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.872778 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33758,"thread_start_us":32804,"threads_started":1}
I20260929 00:54:54.874128 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.809503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39627 (local address 127.0.97.61:34477)
0929 00:54:54.810010 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.810019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.843498 (+ 33479us) server_negotiation.cc:407] Connection header received
0929 00:54:54.843665 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.843673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.843790 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.843916 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:54.852668 (+ 8752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.853726 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.855615 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.856053 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.856195 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.861721 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:54.861728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:54.861733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:54.861781 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:54.865423 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.867712 (+ 2289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.867717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.867721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.867864 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:54.868598 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:54.868603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:54.868605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:54.868919 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:54.869082 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.873499 (+ 4417us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.873731 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260929 00:54:54.874851 7779 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34477
I20260929 00:54:54.875180 7779 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.876063 7779 heartbeater.cc:511] Master 127.0.97.61:34477 requested a full tablet report, sending...
I20260929 00:54:54.876653 7777 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37443
I20260929 00:54:54.876821 7777 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.877295 7777 heartbeater.cc:511] Master 127.0.97.60:37443 requested a full tablet report, sending...
I20260929 00:54:54.877911 7121 ts_manager.cc:194] Registered new tserver with Master: e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:54.878211 7778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46767
I20260929 00:54:54.878324 7778 heartbeater.cc:464] Registering TS with master...
I20260929 00:54:54.878367 7311 ts_manager.cc:194] Registered new tserver with Master: e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:54.878731 7778 heartbeater.cc:511] Master 127.0.97.62:46767 requested a full tablet report, sending...
I20260929 00:54:54.879660 6523 ts_manager.cc:194] Re-registered known tserver with Master: e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:54.880556 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51625
I20260929 00:54:54.873207 6522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:54.883459 6522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:54.923461 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.907359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45467 (local address 127.0.0.1:59038)
0929 00:54:54.907976 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.907998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.908133 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.913826 (+ 5693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.913835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.913871 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:54.914265 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.914280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.917103 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.917113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.920004 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.920024 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.922550 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.922555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.922745 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.922781 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.922886 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.923037 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
I20260929 00:54:54.923914 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.907591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.0.97.2:45467)
0929 00:54:54.913007 (+ 5416us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.913020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.913058 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:54:54.913200 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.913222 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.913573 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.913834 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:54.914516 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.916887 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.920289 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.922310 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.922482 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.922751 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.922964 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.923189 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5207,"thread_start_us":5064,"threads_started":1}
I20260929 00:54:54.927663 7570 tablet_service.cc:1467] Processing CreateTablet for tablet e2285f78fe884edca0662c17c3fb2aed (DEFAULT_TABLE table=test-table [id=482efc48938e4171b34675f45f583ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:54.929163 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285f78fe884edca0662c17c3fb2aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:54.930588 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.912776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35065 (local address 127.0.0.1:43510)
0929 00:54:54.920796 (+ 8020us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.920821 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.920959 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.922785 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.922790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.922815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:54.923171 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.923185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.926153 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.926157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.927948 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.927958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.929670 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.929677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.929871 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.929896 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.930088 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.930228 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7859,"thread_start_us":7675,"threads_started":1}
I20260929 00:54:54.930604 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.906996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.0.97.3:35065)
0929 00:54:54.922005 (+ 15009us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.922011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.922037 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:54:54.922112 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.922118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.922306 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.922531 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:54.923364 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.926028 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.928114 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.929401 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.929596 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.929815 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.930102 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.930247 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14855,"thread_start_us":14703,"threads_started":1}
I20260929 00:54:54.933547 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.906203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.0.1:50224)
0929 00:54:54.906763 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:54.906790 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:54.906950 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:54.925610 (+ 18660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:54.925613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:54.925634 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:54.925873 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.925879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.927635 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.927642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.929726 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:54.929740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.931524 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.931530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:54.931699 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.932572 (+ 873us) client_negotiation.cc:769] Sending connection context
0929 00:54:54.933116 (+ 544us) client_negotiation.cc:240] Negotiation successful
0929 00:54:54.933250 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":255,"threads_started":1}
I20260929 00:54:54.933607 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:54.906464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.0.97.1:43427)
0929 00:54:54.925058 (+ 18594us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:54.925067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:54.925095 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:54:54.925187 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:54.925198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:54.925430 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:54.925624 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:54.926034 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.927473 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.929903 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:54.931299 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:54.931424 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:54.931697 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:54.933172 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0929 00:54:54.933331 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18425,"thread_start_us":18312,"threads_started":1}
I20260929 00:54:54.934887 7712 tablet_service.cc:1467] Processing CreateTablet for tablet e2285f78fe884edca0662c17c3fb2aed (DEFAULT_TABLE table=test-table [id=482efc48938e4171b34675f45f583ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:54.936163 7712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285f78fe884edca0662c17c3fb2aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:54.937150 7424 tablet_service.cc:1467] Processing CreateTablet for tablet e2285f78fe884edca0662c17c3fb2aed (DEFAULT_TABLE table=test-table [id=482efc48938e4171b34675f45f583ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:54.938413 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285f78fe884edca0662c17c3fb2aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:54.948052 7813 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Bootstrap starting.
I20260929 00:54:54.949743 7812 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Bootstrap starting.
I20260929 00:54:54.951633 7813 tablet_bootstrap.cc:654] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:54.952917 7813 log.cc:824] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:54.953393 7812 tablet_bootstrap.cc:654] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:54.954357 7814 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Bootstrap starting.
I20260929 00:54:54.954612 7812 log.cc:824] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:54.957170 7813 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: No bootstrap required, opened a new log
I20260929 00:54:54.957341 7812 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: No bootstrap required, opened a new log
I20260929 00:54:54.957463 7813 ts_tablet_manager.cc:1397] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 00:54:54.957636 7812 ts_tablet_manager.cc:1397] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260929 00:54:54.958623 7814 tablet_bootstrap.cc:654] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:54.959748 7814 log.cc:824] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Log is configured to *not* fsync() on all Append() calls
I20260929 00:54:54.962626 7814 tablet_bootstrap.cc:492] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: No bootstrap required, opened a new log
I20260929 00:54:54.962873 7814 ts_tablet_manager.cc:1397] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 00:54:54.966115 7813 raft_consensus.cc:348] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.966477 7813 raft_consensus.cc:374] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:54.966599 7813 raft_consensus.cc:729] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a3178fdc8c49b2a208caad63034b98, State: Initialized, Role: FOLLOWER
I20260929 00:54:54.967265 7813 consensus_queue.cc:260] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.968143 7812 raft_consensus.cc:348] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.968568 7812 raft_consensus.cc:374] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:54.968659 7812 raft_consensus.cc:729] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87284e1bb781498db54719968ba55cd8, State: Initialized, Role: FOLLOWER
I20260929 00:54:54.968991 7813 ts_tablet_manager.cc:1428] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 00:54:54.969244 7778 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
I20260929 00:54:54.969298 7812 consensus_queue.cc:260] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.976866 7814 raft_consensus.cc:348] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.977334 7812 ts_tablet_manager.cc:1428] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260929 00:54:54.977360 7814 raft_consensus.cc:374] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:54.977447 7814 raft_consensus.cc:729] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f9db9faab8437e8608c109ee3e9837, State: Initialized, Role: FOLLOWER
I20260929 00:54:54.977663 7636 heartbeater.cc:503] Master 127.0.97.62:46767 was elected leader, sending a full tablet report...
I20260929 00:54:54.978096 7814 consensus_queue.cc:260] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:54.980834 7814 ts_tablet_manager.cc:1428] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
W20260929 00:54:55.095852 7788 tablet.cc:2369] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:54:55.125821 7492 tablet.cc:2369] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:55.222743 7819 raft_consensus.cc:482] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:55.223032 7819 raft_consensus.cc:504] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
W20260929 00:54:55.224901 7638 tablet.cc:2369] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:54:55.225409 7819 leader_election.cc:288] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427), e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:55.232144 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.225856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43201 (local address 127.0.97.1:43427)
0929 00:54:55.226110 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.226130 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.226839 (+ 709us) server_negotiation.cc:407] Connection header received
0929 00:54:55.227074 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.227078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.227144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.227245 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:55.227847 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.228613 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.230475 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.231296 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.231439 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.231511 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.231869 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.231962 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 00:54:55.232574 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.225626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.97.2:43201)
0929 00:54:55.226684 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:55.226732 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:55.226916 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:55.227274 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:55.227278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:55.227308 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:55.227692 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:55.227701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.228743 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.228749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:55.230345 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:55.230356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.231517 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.231523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:55.231703 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.231749 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 00:54:55.231852 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 00:54:55.231978 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":1573,"threads_started":1}
I20260929 00:54:55.233048 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.226078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50473 (local address 127.0.97.3:35065)
0929 00:54:55.226280 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.226288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.227543 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 00:54:55.227731 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.227736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.227802 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.227896 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:55.228367 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.229159 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.230965 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.232121 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.232263 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.232370 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.232691 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.232826 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 00:54:55.233098 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.225641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35065 (local address 127.0.97.2:50473)
0929 00:54:55.227437 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:55.227451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:55.227587 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:55.227922 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:55.227926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:55.227948 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:55.228228 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:55.228236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.229294 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.229300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:55.230819 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:55.230830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.232334 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.232340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:55.232473 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.232537 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 00:54:55.232626 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 00:54:55.232777 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1678,"thread_start_us":640,"threads_started":1}
I20260929 00:54:55.233597 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285f78fe884edca0662c17c3fb2aed" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f9db9faab8437e8608c109ee3e9837" is_pre_election: true
I20260929 00:54:55.233911 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285f78fe884edca0662c17c3fb2aed" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3178fdc8c49b2a208caad63034b98" is_pre_election: true
I20260929 00:54:55.234118 7444 raft_consensus.cc:2454] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 0.
I20260929 00:54:55.234442 7732 raft_consensus.cc:2454] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 0.
I20260929 00:54:55.235255 7526 leader_election.cc:302] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f9db9faab8437e8608c109ee3e9837, 87284e1bb781498db54719968ba55cd8; no voters:
I20260929 00:54:55.235667 7819 raft_consensus.cc:2781] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:55.235819 7819 raft_consensus.cc:482] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:55.235924 7819 raft_consensus.cc:3037] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:55.238179 7819 raft_consensus.cc:504] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:55.238802 7819 leader_election.cc:288] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 election: Requested vote from peers 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427), e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:55.239148 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285f78fe884edca0662c17c3fb2aed" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f9db9faab8437e8608c109ee3e9837"
I20260929 00:54:55.239276 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285f78fe884edca0662c17c3fb2aed" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3178fdc8c49b2a208caad63034b98"
I20260929 00:54:55.239370 7444 raft_consensus.cc:3037] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:55.239500 7732 raft_consensus.cc:3037] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:55.242595 7732 raft_consensus.cc:2454] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 1.
I20260929 00:54:55.242595 7444 raft_consensus.cc:2454] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 1.
I20260929 00:54:55.242990 7523 leader_election.cc:302] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87284e1bb781498db54719968ba55cd8, e4a3178fdc8c49b2a208caad63034b98; no voters:
I20260929 00:54:55.243346 7819 raft_consensus.cc:2781] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:55.243793 7819 raft_consensus.cc:686] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 87284e1bb781498db54719968ba55cd8, State: Running, Role: LEADER
I20260929 00:54:55.244311 7819 consensus_queue.cc:237] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } }
I20260929 00:54:55.249972 6523 catalog_manager.cc:5495] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87284e1bb781498db54719968ba55cd8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "87284e1bb781498db54719968ba55cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } health_report { overall_health: HEALTHY } } }
I20260929 00:54:55.283747 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.278794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.97.2:45467)
0929 00:54:55.278965 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.278969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.279053 (+ 84us) server_negotiation.cc:407] Connection header received
0929 00:54:55.279239 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.279243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.279314 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.279432 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:55.279774 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.280445 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.281963 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.282332 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.282424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.282817 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:55.283316 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:55.283423 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.283492 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.283548 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:54:55.291467 7732 raft_consensus.cc:1261] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Refusing update from remote peer 87284e1bb781498db54719968ba55cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:55.291476 7444 raft_consensus.cc:1261] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Refusing update from remote peer 87284e1bb781498db54719968ba55cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:55.292459 7823 consensus_queue.cc:1035] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:55.292866 7819 consensus_queue.cc:1035] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:55.307574 7828 mvcc.cc:205] Tried to move back new op lower bound from 7334474937504378880 to 7334474937324974080. Current Snapshot: MvccSnapshot[applied={T|T < 7334474937504378880}]
I20260929 00:54:55.321408 6522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:55.321702 6522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:55.327466 6522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfad066ad17245efbe1c29c65730bb11: Sending DeleteTablet for 3 replicas of tablet e2285f78fe884edca0662c17c3fb2aed
I20260929 00:54:55.328435 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285f78fe884edca0662c17c3fb2aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260929 00:54:55.328980 7712 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285f78fe884edca0662c17c3fb2aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43510
I20260929 00:54:55.330179 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285f78fe884edca0662c17c3fb2aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:55 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59038
I20260929 00:54:55.331528 7836 tablet_replica.cc:330] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: stopping tablet replica
I20260929 00:54:55.332041 7836 raft_consensus.cc:2229] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:55.334955 7836 raft_consensus.cc:2258] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:55.337752 7836 ts_tablet_manager.cc:1905] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:55.347546 7836 ts_tablet_manager.cc:1918] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:55.347729 7836 log.cc:1196] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/e2285f78fe884edca0662c17c3fb2aed
I20260929 00:54:55.348326 7836 ts_tablet_manager.cc:1939] T e2285f78fe884edca0662c17c3fb2aed P e4a3178fdc8c49b2a208caad63034b98: Deleting consensus metadata
I20260929 00:54:55.350529 6507 catalog_manager.cc:4841] TS e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065): tablet e2285f78fe884edca0662c17c3fb2aed (table test-table [id=482efc48938e4171b34675f45f583ddf]) successfully deleted
I20260929 00:54:55.354079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6491
I20260929 00:54:55.354064 7837 tablet_replica.cc:330] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: stopping tablet replica
I20260929 00:54:55.355348 6553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:55.360623 7837 raft_consensus.cc:2229] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:55.361506 7837 raft_consensus.cc:2258] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:55.362442 7835 tablet_replica.cc:330] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: stopping tablet replica
I20260929 00:54:55.362865 7835 raft_consensus.cc:2229] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:55.363358 7835 raft_consensus.cc:2258] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:55.364018 7837 ts_tablet_manager.cc:1905] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:55.370995 7835 ts_tablet_manager.cc:1905] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:55.378742 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.332802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.0.97.61:34477)
0929 00:54:55.333012 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.333020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.354407 (+ 21387us) server_negotiation.cc:407] Connection header received
0929 00:54:55.356996 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.357006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.357147 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.357243 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:55.363764 (+ 6521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.364974 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.366914 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.367529 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.367630 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.371303 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:55.371310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:55.371316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:55.371354 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:55.375612 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.376441 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.376448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.376451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.376630 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.377053 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.377058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.377061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.377407 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:55.377637 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.378166 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.378377 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:54:55.378896 7837 ts_tablet_manager.cc:1918] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:55.379068 7837 log.cc:1196] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/e2285f78fe884edca0662c17c3fb2aed
I20260929 00:54:55.379649 7837 ts_tablet_manager.cc:1939] T e2285f78fe884edca0662c17c3fb2aed P 87284e1bb781498db54719968ba55cd8: Deleting consensus metadata
I20260929 00:54:55.382787 7835 ts_tablet_manager.cc:1918] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:55.382946 7835 log.cc:1196] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/e2285f78fe884edca0662c17c3fb2aed
I20260929 00:54:55.383499 7835 ts_tablet_manager.cc:1939] T e2285f78fe884edca0662c17c3fb2aed P 26f9db9faab8437e8608c109ee3e9837: Deleting consensus metadata
I20260929 00:54:55.394765 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.353400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.0.97.60:37443)
0929 00:54:55.353615 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.353623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.380977 (+ 27354us) server_negotiation.cc:407] Connection header received
0929 00:54:55.381226 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.381232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.381316 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.381387 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:55.383233 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.384316 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.386481 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.386870 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.387005 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.389501 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:55.389507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:55.389512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:55.389557 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:55.392157 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.392833 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.392839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.392844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.393009 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.393371 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.393376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.393380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.393668 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:55.393904 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.394220 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.394430 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:54:55.513382 6510 catalog_manager.cc:4841] TS 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427): tablet e2285f78fe884edca0662c17c3fb2aed (table test-table [id=482efc48938e4171b34675f45f583ddf]) successfully deleted
I20260929 00:54:55.514786 6508 catalog_manager.cc:4841] TS 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467): tablet e2285f78fe884edca0662c17c3fb2aed (table test-table [id=482efc48938e4171b34675f45f583ddf]) successfully deleted
I20260929 00:54:55.515517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6491
I20260929 00:54:55.564033 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.541431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.97.61:34477)
0929 00:54:55.541615 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.541621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.546307 (+ 4686us) server_negotiation.cc:407] Connection header received
0929 00:54:55.546610 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.546625 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.546720 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.546808 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:55.550539 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.551678 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.553702 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.554367 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.554498 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.557997 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:55.558004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:55.558008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:55.558045 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:55.560962 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.561575 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.561581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.561584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.561730 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.562208 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.562213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.562218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.562544 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:55.562796 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.563326 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.563699 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:54:55.565496 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:55.541183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.97.60:37443)
0929 00:54:55.541397 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:55.541403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:55.550305 (+ 8902us) server_negotiation.cc:407] Connection header received
0929 00:54:55.550570 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:55.550575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:55.550670 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:55.550728 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:55.552377 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.553156 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.554885 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:55.555461 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:55.555602 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:55.558242 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:55.558249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:55.558255 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:55.558294 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:55.561651 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.562411 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.562416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.562420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.562543 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:55.562952 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:55.562960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:55.562966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:55.563327 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:55.563519 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:55.564049 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 00:54:55.565264 (+ 1215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:54:56.316543 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.315703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:40455)
0929 00:54:56.316170 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.316306 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260929 00:54:56.316768 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.315712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:52129)
0929 00:54:56.316391 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.316584 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":471,"threads_started":1}
W20260929 00:54:56.317087 7636 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
W20260929 00:54:56.317238 7490 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.320236 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.319444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:41227)
0929 00:54:56.319866 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.320004 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":238,"threads_started":1}
W20260929 00:54:56.320609 7778 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46767 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.829013 7850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfad066ad17245efbe1c29c65730bb11)
I20260929 00:54:56.829274 7850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:56.829977 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bfad066ad17245efbe1c29c65730bb11)
I20260929 00:54:56.830199 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:56.830871 7850 leader_election.cc:288] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60:37443)
I20260929 00:54:56.837803 7851 leader_election.cc:288] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61:34477)
I20260929 00:54:56.841277 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.831093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44172)
0929 00:54:56.840745 (+ 9652us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.840949 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9486,"thread_start_us":9308,"threads_started":1}
W20260929 00:54:56.841694 7297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.842710 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.830964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44166)
0929 00:54:56.841083 (+ 10119us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.841289 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9976,"thread_start_us":10860,"threads_started":1}
W20260929 00:54:56.842986 7107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.845058 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.844700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44188)
0929 00:54:56.844827 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.844910 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 00:54:56.845356 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.846172 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.844380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44184)
0929 00:54:56.845802 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.845958 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1221}
W20260929 00:54:56.846302 7297 leader_election.cc:334] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.850162 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.831857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37443 (local address 127.0.0.1:57736)
0929 00:54:56.842837 (+ 10980us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.842852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:56.842999 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:56.844866 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:56.844872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:56.844895 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:56.845223 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:56.845235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.846794 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.846802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:56.848105 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:56.848116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.849253 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.849259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:56.849437 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:56.849525 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 00:54:56.849654 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 00:54:56.849778 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10854,"mutex_wait_us":40,"thread_start_us":10543,"threads_started":1}
I20260929 00:54:56.850199 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.831564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.0.97.60:37443)
0929 00:54:56.844481 (+ 12917us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:56.844488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:56.844535 (+ 47us) server_negotiation.cc:407] Connection header received
0929 00:54:56.844595 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:56.844599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:56.844670 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:56.844769 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:56.845417 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.846652 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.848258 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.849084 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.849232 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:56.849323 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:56.849679 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 00:54:56.849798 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":259,"server-negotiator.queue_time_us":12794,"spinlock_wait_cycles":3456,"thread_start_us":12765,"threads_started":1}
I20260929 00:54:56.850375 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.831257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34477 (local address 127.0.0.1:56864)
0929 00:54:56.841519 (+ 10262us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.841540 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:56.841708 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:56.844654 (+ 2946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:56.844659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:56.844688 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:56.844953 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:56.844964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.846814 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.846819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:56.848339 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:56.848351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.849498 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.849504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:56.849693 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:56.849716 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:54:56.849822 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:54:56.849962 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10149,"mutex_wait_us":3,"thread_start_us":10902,"threads_started":1}
I20260929 00:54:56.850703 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.842654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.0.97.61:34477)
0929 00:54:56.844325 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:56.844332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:56.844350 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:54:56.844404 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:56.844410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:56.844478 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:56.845289 (+ 811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:56.845544 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.846674 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.848488 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:56.849322 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:56.849512 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:56.849620 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:56.849842 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 00:54:56.849966 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1542,"thread_start_us":1340,"threads_started":1}
I20260929 00:54:56.850911 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05a723e6d0384131b6a81a2520b9d979" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0768315e14c4cab9beb8fc2bdb59238" is_pre_election: true
I20260929 00:54:56.851229 7321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a723e6d0384131b6a81a2520b9d979 in term 1.
I20260929 00:54:56.851202 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0768315e14c4cab9beb8fc2bdb59238" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "05a723e6d0384131b6a81a2520b9d979" is_pre_election: true
I20260929 00:54:56.851423 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0768315e14c4cab9beb8fc2bdb59238 in term 1.
I20260929 00:54:56.851686 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05a723e6d0384131b6a81a2520b9d979, f0768315e14c4cab9beb8fc2bdb59238; no voters: bfad066ad17245efbe1c29c65730bb11
I20260929 00:54:56.852001 7850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 00:54:56.852128 7850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfad066ad17245efbe1c29c65730bb11)
I20260929 00:54:56.852236 7850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:54:56.852349 7295 leader_election.cc:302] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05a723e6d0384131b6a81a2520b9d979, f0768315e14c4cab9beb8fc2bdb59238; no voters: bfad066ad17245efbe1c29c65730bb11
I20260929 00:54:56.852679 7851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 00:54:56.852794 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bfad066ad17245efbe1c29c65730bb11)
I20260929 00:54:56.852905 7851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:54:56.854560 7850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:56.854902 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:56.855264 7850 leader_election.cc:288] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 election: Requested vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60:37443)
I20260929 00:54:56.855516 7851 leader_election.cc:288] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 election: Requested vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), 05a723e6d0384131b6a81a2520b9d979 (127.0.97.61:34477)
I20260929 00:54:56.855787 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.855306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44198)
0929 00:54:56.855526 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.855610 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 00:54:56.855726 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05a723e6d0384131b6a81a2520b9d979" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0768315e14c4cab9beb8fc2bdb59238"
I20260929 00:54:56.856004 7321 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05a723e6d0384131b6a81a2520b9d979 in current term 2: Already voted for candidate f0768315e14c4cab9beb8fc2bdb59238 in this term.
I20260929 00:54:56.856285 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.855818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44204)
0929 00:54:56.856027 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.856112 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 00:54:56.856750 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.856302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44218)
0929 00:54:56.856440 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.856557 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 00:54:56.856716 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0768315e14c4cab9beb8fc2bdb59238" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "05a723e6d0384131b6a81a2520b9d979"
I20260929 00:54:56.856992 7131 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0768315e14c4cab9beb8fc2bdb59238 in current term 2: Already voted for candidate 05a723e6d0384131b6a81a2520b9d979 in this term.
W20260929 00:54:56.857208 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.857236 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:56.856873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44220)
0929 00:54:56.857007 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:56.857106 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 00:54:56.857359 7107 leader_election.cc:302] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05a723e6d0384131b6a81a2520b9d979; no voters: bfad066ad17245efbe1c29c65730bb11, f0768315e14c4cab9beb8fc2bdb59238
W20260929 00:54:56.857429 7297 leader_election.cc:334] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:56.857630 7297 leader_election.cc:302] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0768315e14c4cab9beb8fc2bdb59238; no voters: 05a723e6d0384131b6a81a2520b9d979, bfad066ad17245efbe1c29c65730bb11
I20260929 00:54:56.857656 7850 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 00:54:56.857892 7851 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 00:54:57.319859 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:57.317984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:58739)
0929 00:54:57.319571 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:57.319684 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1440,"thread_start_us":1320,"threads_started":1}
I20260929 00:54:57.320022 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:57.317841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:43847)
0929 00:54:57.319753 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:57.319843 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1812,"thread_start_us":1692,"threads_started":1}
I20260929 00:54:57.322079 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:57.321308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:36515)
0929 00:54:57.321761 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:57.321870 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":254,"threads_started":1}
I20260929 00:54:58.321699 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.320749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:56531)
0929 00:54:58.321363 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.321467 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":404,"threads_started":1}
I20260929 00:54:58.321672 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.320857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:51553)
0929 00:54:58.321363 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.321476 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260929 00:54:58.323557 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.322929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:33545)
0929 00:54:58.323341 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.323423 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":221,"threads_started":1}
I20260929 00:54:58.811733 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:58.811931 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:58.812587 7866 leader_election.cc:288] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60:37443)
I20260929 00:54:58.813279 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05a723e6d0384131b6a81a2520b9d979" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0768315e14c4cab9beb8fc2bdb59238" is_pre_election: true
I20260929 00:54:58.813511 7321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a723e6d0384131b6a81a2520b9d979 in term 2.
I20260929 00:54:58.813629 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.812793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44496)
0929 00:54:58.813262 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.813351 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":209,"threads_started":1}
I20260929 00:54:58.814004 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a723e6d0384131b6a81a2520b9d979, f0768315e14c4cab9beb8fc2bdb59238; no voters:
I20260929 00:54:58.814323 7866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 00:54:58.814432 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:58.814590 7866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:54:58.815143 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.814724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44504)
0929 00:54:58.814899 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.814987 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 00:54:58.815331 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:58.817494 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:58.818069 7866 leader_election.cc:288] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 election: Requested vote from peers bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767), f0768315e14c4cab9beb8fc2bdb59238 (127.0.97.60:37443)
I20260929 00:54:58.818585 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.818129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44518)
0929 00:54:58.818342 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.818438 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 00:54:58.818534 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05a723e6d0384131b6a81a2520b9d979" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0768315e14c4cab9beb8fc2bdb59238"
I20260929 00:54:58.818771 7321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:54:58.820824 7321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a723e6d0384131b6a81a2520b9d979 in term 3.
I20260929 00:54:58.821218 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a723e6d0384131b6a81a2520b9d979, f0768315e14c4cab9beb8fc2bdb59238; no voters:
I20260929 00:54:58.821380 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.820929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44534)
0929 00:54:58.821117 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.821231 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:54:58.821501 7866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 3 FOLLOWER]: Leader election won for term 3
W20260929 00:54:58.821596 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
I20260929 00:54:58.821919 7866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [term 3 LEADER]: Becoming Leader. State: Replica: 05a723e6d0384131b6a81a2520b9d979, State: Running, Role: LEADER
I20260929 00:54:58.822307 7866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } }
I20260929 00:54:58.824815 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05a723e6d0384131b6a81a2520b9d979. Latest consensus state: current_term: 3 leader_uuid: "05a723e6d0384131b6a81a2520b9d979" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:58.825043 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:58.825632 7871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:54:58.828969 7871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:54:58.829834 7871 catalog_manager.cc:1259] Loaded cluster ID: 92e065524b9d4501af31b5d644ca9324
I20260929 00:54:58.829931 7871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:54:58.831152 7871 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:54:58.831931 7871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Loaded TSK: 0
I20260929 00:54:58.832808 7871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:54:58.915839 7121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:54:58.922626 7121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:54:58.926836 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.926369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44568)
0929 00:54:58.926572 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.926647 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 00:54:58.927157 7321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [term 3 FOLLOWER]: Refusing update from remote peer 05a723e6d0384131b6a81a2520b9d979: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 00:54:58.927780 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.927448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44580)
0929 00:54:58.927611 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.927671 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 00:54:58.927822 7866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 00:54:58.928134 7107 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 -> Peer bfad066ad17245efbe1c29c65730bb11 (127.0.97.62:46767): Couldn't send request to peer bfad066ad17245efbe1c29c65730bb11. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 00:54:58.931159 7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05a723e6d0384131b6a81a2520b9d979. Latest consensus state: current_term: 3 leader_uuid: "05a723e6d0384131b6a81a2520b9d979" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:58.931479 7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:58.931841 7872 mvcc.cc:205] Tried to move back new op lower bound from 7334474952397529088 to 7334474951979716608. Current Snapshot: MvccSnapshot[applied={T|T < 7334474952397529088}]
I20260929 00:54:58.932204 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "05a723e6d0384131b6a81a2520b9d979" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:58.932577 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979 [sys.catalog]: This master's current role is: LEADER
I20260929 00:54:58.932852 7874 mvcc.cc:205] Tried to move back new op lower bound from 7334474952397529088 to 7334474951979716608. Current Snapshot: MvccSnapshot[applied={T|T < 7334474952397529088}]
I20260929 00:54:58.939930 7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "05a723e6d0384131b6a81a2520b9d979" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bfad066ad17245efbe1c29c65730bb11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46767 } } peers { permanent_uuid: "05a723e6d0384131b6a81a2520b9d979" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34477 } attrs { promote: false } } peers { permanent_uuid: "f0768315e14c4cab9beb8fc2bdb59238" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37443 } attrs { promote: false } } }
I20260929 00:54:58.940174 7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0768315e14c4cab9beb8fc2bdb59238 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:54:58.963560 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.944122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43427 (local address 127.0.0.1:50232)
0929 00:54:58.944261 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.944277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:58.944367 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:58.945065 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:58.945070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:58.945095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:58.945411 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.945424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.958155 (+ 12731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.958164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.960906 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.960923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.962783 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.962788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.962950 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.962984 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:54:58.963074 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 00:54:58.963216 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 00:54:58.965154 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.944306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.0.97.1:43427)
0929 00:54:58.944753 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:58.944759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:58.944777 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:54:58.944841 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:58.944846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:58.944916 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:58.953049 (+ 8133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:58.953306 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.954758 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.961189 (+ 6431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.962602 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.962767 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.962915 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:58.964259 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0929 00:54:58.964808 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":238,"threads_started":1}
I20260929 00:54:58.965968 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 88bded114da84c63aa5ea21f5c4471cf (DEFAULT_TABLE table=test-table [id=6400c802989a4713a7e4c48512766f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:58.966409 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bded114da84c63aa5ea21f5c4471cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:58.969245 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.944960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45467 (local address 127.0.0.1:59056)
0929 00:54:58.960857 (+ 15897us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.960880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:58.961015 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:58.962115 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:58.962126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:58.962146 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:58.962450 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.962467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.963892 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.963898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.966532 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.966545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.968271 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.968276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.968520 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.968552 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:54:58.968655 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 00:54:58.968783 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15769,"thread_start_us":15637,"threads_started":1}
I20260929 00:54:58.969991 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.945038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.97.2:45467)
0929 00:54:58.961807 (+ 16769us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:58.961814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:58.961833 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:54:58.961915 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:58.961922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:58.961996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:58.962095 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:58.962655 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.963715 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.966719 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.967944 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.968063 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.968155 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:58.969550 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0929 00:54:58.969701 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16278,"thread_start_us":16169,"threads_started":1}
I20260929 00:54:58.971078 7570 tablet_service.cc:1467] Processing CreateTablet for tablet 88bded114da84c63aa5ea21f5c4471cf (DEFAULT_TABLE table=test-table [id=6400c802989a4713a7e4c48512766f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:58.971689 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bded114da84c63aa5ea21f5c4471cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:58.973294 7884 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Bootstrap starting.
I20260929 00:54:58.975342 7884 tablet_bootstrap.cc:654] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:58.980819 7886 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Bootstrap starting.
I20260929 00:54:58.982324 7886 tablet_bootstrap.cc:654] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:58.984666 7884 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: No bootstrap required, opened a new log
I20260929 00:54:58.984807 7884 ts_tablet_manager.cc:1397] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260929 00:54:58.985003 7886 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: No bootstrap required, opened a new log
I20260929 00:54:58.985145 7886 ts_tablet_manager.cc:1397] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 00:54:58.985599 7884 raft_consensus.cc:348] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:58.985857 7884 raft_consensus.cc:374] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:58.985944 7884 raft_consensus.cc:729] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f9db9faab8437e8608c109ee3e9837, State: Initialized, Role: FOLLOWER
I20260929 00:54:58.985911 7886 raft_consensus.cc:348] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:58.986181 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.945380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35065 (local address 127.0.0.1:43516)
0929 00:54:58.963931 (+ 18551us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:58.963954 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:54:58.964103 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:54:58.978848 (+ 14745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:54:58.978857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:54:58.978896 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:54:58.979127 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.979141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.981092 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.981101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.983613 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:54:58.983628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.985406 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.985412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:54:58.985586 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.985621 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:54:58.985738 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 00:54:58.985871 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18400,"thread_start_us":35138,"threads_started":1}
I20260929 00:54:58.986203 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.0.97.3:35065)
0929 00:54:58.978509 (+ 32731us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:58.978517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:58.978538 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:54:58.978606 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:58.978611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:58.978684 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:58.978793 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:54:58.979351 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.980878 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.983911 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:58.985254 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:58.985376 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:58.985470 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:58.985767 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 00:54:58.985907 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32596,"thread_start_us":33269,"threads_started":1}
I20260929 00:54:58.986187 7886 raft_consensus.cc:374] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:58.986300 7886 raft_consensus.cc:729] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87284e1bb781498db54719968ba55cd8, State: Initialized, Role: FOLLOWER
I20260929 00:54:58.986172 7884 consensus_queue.cc:260] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:58.986522 7886 consensus_queue.cc:260] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:58.987449 7712 tablet_service.cc:1467] Processing CreateTablet for tablet 88bded114da84c63aa5ea21f5c4471cf (DEFAULT_TABLE table=test-table [id=6400c802989a4713a7e4c48512766f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:54:58.987882 7712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bded114da84c63aa5ea21f5c4471cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:54:58.994335 7886 ts_tablet_manager.cc:1428] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260929 00:54:58.996915 7884 ts_tablet_manager.cc:1428] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260929 00:54:59.000922 7637 heartbeater.cc:503] Master 127.0.97.61:34477 was elected leader, sending a full tablet report...
I20260929 00:54:59.002105 7491 heartbeater.cc:503] Master 127.0.97.61:34477 was elected leader, sending a full tablet report...
I20260929 00:54:59.003031 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.990940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:55229)
0929 00:54:59.002166 (+ 11226us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.002349 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11038,"thread_start_us":10914,"threads_started":1}
I20260929 00:54:59.003602 7891 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Bootstrap starting.
I20260929 00:54:59.004343 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.003964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:37663)
0929 00:54:59.004151 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.004222 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 00:54:59.005841 7891 tablet_bootstrap.cc:654] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Neither blocks nor log segments found. Creating new log.
I20260929 00:54:59.009116 7891 tablet_bootstrap.cc:492] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: No bootstrap required, opened a new log
I20260929 00:54:59.009305 7891 ts_tablet_manager.cc:1397] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 00:54:59.010376 7891 raft_consensus.cc:348] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:59.010684 7891 raft_consensus.cc:374] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:54:59.010794 7891 raft_consensus.cc:729] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a3178fdc8c49b2a208caad63034b98, State: Initialized, Role: FOLLOWER
I20260929 00:54:59.011118 7891 consensus_queue.cc:260] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:59.012058 7891 ts_tablet_manager.cc:1428] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:54:59.012598 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:58.995921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:48629)
0929 00:54:59.012167 (+ 16246us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.012285 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16109,"threads_started":1}
I20260929 00:54:59.014410 7779 heartbeater.cc:503] Master 127.0.97.61:34477 was elected leader, sending a full tablet report...
I20260929 00:54:59.015220 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.014651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:51787)
0929 00:54:59.014944 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.015056 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 00:54:59.016412 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.012830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:60395)
0929 00:54:59.016120 (+ 3290us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.016215 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3175,"thread_start_us":3038,"threads_started":1}
I20260929 00:54:59.017316 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.017002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:39843)
0929 00:54:59.017150 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.017208 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:54:59.042521 7889 raft_consensus.cc:482] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:54:59.042762 7889 raft_consensus.cc:504] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:59.043679 7889 leader_election.cc:288] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427), e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:59.048630 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bded114da84c63aa5ea21f5c4471cf" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f9db9faab8437e8608c109ee3e9837" is_pre_election: true
I20260929 00:54:59.049001 7443 raft_consensus.cc:2454] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 0.
I20260929 00:54:59.049082 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bded114da84c63aa5ea21f5c4471cf" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3178fdc8c49b2a208caad63034b98" is_pre_election: true
I20260929 00:54:59.049307 7732 raft_consensus.cc:2454] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 0.
I20260929 00:54:59.049700 7526 leader_election.cc:302] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f9db9faab8437e8608c109ee3e9837, 87284e1bb781498db54719968ba55cd8; no voters:
I20260929 00:54:59.050149 7889 raft_consensus.cc:2781] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:54:59.050283 7889 raft_consensus.cc:482] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:54:59.050411 7889 raft_consensus.cc:3037] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:59.052439 7889 raft_consensus.cc:504] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:59.053066 7889 leader_election.cc:288] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 election: Requested vote from peers 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427), e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065)
I20260929 00:54:59.053632 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bded114da84c63aa5ea21f5c4471cf" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3178fdc8c49b2a208caad63034b98"
I20260929 00:54:59.053635 7443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bded114da84c63aa5ea21f5c4471cf" candidate_uuid: "87284e1bb781498db54719968ba55cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f9db9faab8437e8608c109ee3e9837"
I20260929 00:54:59.053830 7732 raft_consensus.cc:3037] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:59.053831 7443 raft_consensus.cc:3037] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:54:59.056802 7443 raft_consensus.cc:2454] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 1.
I20260929 00:54:59.056802 7732 raft_consensus.cc:2454] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87284e1bb781498db54719968ba55cd8 in term 1.
I20260929 00:54:59.057204 7523 leader_election.cc:302] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26f9db9faab8437e8608c109ee3e9837, 87284e1bb781498db54719968ba55cd8, e4a3178fdc8c49b2a208caad63034b98; no voters:
I20260929 00:54:59.057492 7889 raft_consensus.cc:2781] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:54:59.057884 7889 raft_consensus.cc:686] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 87284e1bb781498db54719968ba55cd8, State: Running, Role: LEADER
I20260929 00:54:59.058210 7889 consensus_queue.cc:237] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260929 00:54:59.058929 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.058495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:53517)
0929 00:54:59.058634 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.058759 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 00:54:59.060819 7121 catalog_manager.cc:5495] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87284e1bb781498db54719968ba55cd8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "87284e1bb781498db54719968ba55cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87284e1bb781498db54719968ba55cd8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } health_report { overall_health: UNKNOWN } } }
I20260929 00:54:59.084131 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.079773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.97.2:45467)
0929 00:54:59.079944 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:59.079949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:59.079965 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:54:59.080092 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:59.080095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:59.080159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:59.080234 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:54:59.080630 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.081383 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.082680 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.083091 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.083183 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:59.083521 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:54:59.083625 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:54:59.083691 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:59.083864 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 00:54:59.083928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:54:59.087317 7443 raft_consensus.cc:1261] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Refusing update from remote peer 87284e1bb781498db54719968ba55cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:59.087317 7732 raft_consensus.cc:1261] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Refusing update from remote peer 87284e1bb781498db54719968ba55cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:54:59.088027 7896 consensus_queue.cc:1035] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f9db9faab8437e8608c109ee3e9837" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:59.088344 7889 consensus_queue.cc:1035] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a3178fdc8c49b2a208caad63034b98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:54:59.090485 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.089852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:59425)
0929 00:54:59.090071 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.090238 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 00:54:59.091414 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.090988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:40505)
0929 00:54:59.091166 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.091262 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:54:59.097393 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.096891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.3:54803)
0929 00:54:59.097135 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.097240 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 00:54:59.098541 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.097970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.2:34753)
0929 00:54:59.098230 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.098330 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 00:54:59.099591 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.098701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.97.1:50433)
0929 00:54:59.099363 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.099445 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564}
I20260929 00:54:59.105041 7899 mvcc.cc:205] Tried to move back new op lower bound from 7334474953055346688 to 7334474952944590848. Current Snapshot: MvccSnapshot[applied={T|T < 7334474953055346688 or (T in {7334474953055346688})}]
I20260929 00:54:59.106361 7120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:59.106647 7120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:54:59.112720 7120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05a723e6d0384131b6a81a2520b9d979: Sending DeleteTablet for 3 replicas of tablet 88bded114da84c63aa5ea21f5c4471cf
I20260929 00:54:59.113611 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bded114da84c63aa5ea21f5c4471cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:59 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:59056
I20260929 00:54:59.113746 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bded114da84c63aa5ea21f5c4471cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:59 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50232
I20260929 00:54:59.113997 7712 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bded114da84c63aa5ea21f5c4471cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:59 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43516
I20260929 00:54:59.114030 7711 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bded114da84c63aa5ea21f5c4471cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:54:59 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43516
I20260929 00:54:59.114831 7910 tablet_replica.cc:330] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: stopping tablet replica
I20260929 00:54:59.115149 7910 raft_consensus.cc:2229] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:59.115527 7910 raft_consensus.cc:2258] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:59.116621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 7288
I20260929 00:54:59.116956 7910 ts_tablet_manager.cc:1905] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:59.126281 7910 ts_tablet_manager.cc:1918] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:59.126562 7910 log.cc:1196] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/88bded114da84c63aa5ea21f5c4471cf
I20260929 00:54:59.127360 7910 ts_tablet_manager.cc:1939] T 88bded114da84c63aa5ea21f5c4471cf P e4a3178fdc8c49b2a208caad63034b98: Deleting consensus metadata
I20260929 00:54:59.128952 7105 catalog_manager.cc:4841] TS e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065): tablet 88bded114da84c63aa5ea21f5c4471cf (table test-table [id=6400c802989a4713a7e4c48512766f38]) successfully deleted
W20260929 00:54:59.130069 7105 catalog_manager.cc:4808] TS e4a3178fdc8c49b2a208caad63034b98 (127.0.97.3:35065): delete failed for tablet 88bded114da84c63aa5ea21f5c4471cf because the tablet was not found. No further retry: Not found: Tablet not found: 88bded114da84c63aa5ea21f5c4471cf
I20260929 00:54:59.132252 7909 tablet_replica.cc:330] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: stopping tablet replica
I20260929 00:54:59.132545 7909 raft_consensus.cc:2229] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:54:59.132953 7908 tablet_replica.cc:330] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: stopping tablet replica
I20260929 00:54:59.132994 7909 raft_consensus.cc:2258] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:59.133191 7908 raft_consensus.cc:2229] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:54:59.133584 7908 raft_consensus.cc:2258] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:54:59.134333 7909 ts_tablet_manager.cc:1905] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:59.134858 7908 ts_tablet_manager.cc:1905] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:54:59.140005 7908 ts_tablet_manager.cc:1918] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:59.140136 7908 log.cc:1196] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/88bded114da84c63aa5ea21f5c4471cf
I20260929 00:54:59.140381 7909 ts_tablet_manager.cc:1918] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:54:59.140568 7909 log.cc:1196] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/88bded114da84c63aa5ea21f5c4471cf
I20260929 00:54:59.140612 7908 ts_tablet_manager.cc:1939] T 88bded114da84c63aa5ea21f5c4471cf P 87284e1bb781498db54719968ba55cd8: Deleting consensus metadata
I20260929 00:54:59.141072 7909 ts_tablet_manager.cc:1939] T 88bded114da84c63aa5ea21f5c4471cf P 26f9db9faab8437e8608c109ee3e9837: Deleting consensus metadata
I20260929 00:54:59.141846 7106 catalog_manager.cc:4841] TS 87284e1bb781498db54719968ba55cd8 (127.0.97.2:45467): tablet 88bded114da84c63aa5ea21f5c4471cf (table test-table [id=6400c802989a4713a7e4c48512766f38]) successfully deleted
I20260929 00:54:59.142164 7108 catalog_manager.cc:4841] TS 26f9db9faab8437e8608c109ee3e9837 (127.0.97.1:43427): tablet 88bded114da84c63aa5ea21f5c4471cf (table test-table [id=6400c802989a4713a7e4c48512766f38]) successfully deleted
I20260929 00:54:59.145299 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.118735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.0.97.1:43427)
0929 00:54:59.118949 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:59.118956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:59.128510 (+ 9554us) server_negotiation.cc:407] Connection header received
0929 00:54:59.129058 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:59.129067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:59.129176 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:59.129278 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:59.131764 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.132877 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.134664 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.135326 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.135468 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:59.138728 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:59.138746 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:59.138765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:59.138809 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:59.142892 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.143544 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.143551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.143553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.143674 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.143967 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.143970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.143972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.144298 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:59.144642 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:59.144908 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 00:54:59.145082 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 00:54:59.146021 7484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:59.239557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 7288
I20260929 00:54:59.258607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 7501
I20260929 00:54:59.271394 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.259329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.0.97.2:45467)
0929 00:54:59.259499 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:59.259503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:59.259568 (+ 65us) server_negotiation.cc:407] Connection header received
0929 00:54:59.259712 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:59.259715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:59.259785 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:59.259871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:59.261146 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.261930 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.263180 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.263575 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.263665 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:59.266205 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:59.266220 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:59.266238 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:59.266278 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:59.269220 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.269816 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.269823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.269825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.269947 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.270251 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.270254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.270256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.270592 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:59.270807 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:59.270987 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 00:54:59.271138 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:54:59.272135 7630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:59.389343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 7501
I20260929 00:54:59.411037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 7648
I20260929 00:54:59.424180 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.411885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.97.3:35065)
0929 00:54:59.412077 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:54:59.412083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:54:59.412102 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:54:59.412296 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:54:59.412308 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:54:59.412369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:54:59.412478 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:54:59.413812 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.414522 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.415714 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:54:59.416143 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:54:59.416288 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:54:59.418713 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:54:59.418733 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:54:59.418749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:54:59.418784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:54:59.421459 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.422131 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.422139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.422141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.422279 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:54:59.422649 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:54:59.422659 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:54:59.422661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:54:59.422995 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:54:59.423330 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 00:54:59.423656 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 00:54:59.423886 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:54:59.424988 7772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:59.475206 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.474725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44590)
0929 00:54:59.474925 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.475040 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 00:54:59.476565 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:54:59.476154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46767 (local address 127.0.0.1:44602)
0929 00:54:59.476309 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 00:54:59.476397 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:54:59.523123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 7648
I20260929 00:54:59.540683 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 6984
I20260929 00:54:59.541790 7151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:59.661404 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 6984
I20260929 00:54:59.682933 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 7085
I20260929 00:54:59.683947 7341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:54:59.798887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 7085
2026-09-29T00:54:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11450 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:54:59 dist-test-slave-pzsx krb5kdc[7920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:54:59 dist-test-slave-pzsx krb5kdc[7920](info): set up 2 sockets
Sep 29 00:54:59 dist-test-slave-pzsx krb5kdc[7920](info): commencing operation
krb5kdc: starting...
W20260929 00:55:00.889482 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:55:01 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 00:55:01.100234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:01.421576 7933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:01.421885 7933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:01.421972 7933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:01.429260 7933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:01.429361 7933 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 00:55:01.429433 7933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:01.429536 7933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:01.429612 7933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:01.429731 7933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:01.439672 7933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:01.441015 7933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:01.442346 7933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:01.454192 7938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:01.454248 7939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:01.454192 7941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:01.454943 7933 server_base.cc:1034] running on GCE node
I20260929 00:55:01.455687 7933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:01.457145 7933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:01.458361 7933 hybrid_clock.cc:648] HybridClock initialized: now 1790643301458293 us; error 36 us; skew 500 ppm
I20260929 00:55:01.458698 7933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:01 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643301, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:01.461758 7933 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 00:55:01.463209 7933 webserver.cc:460] Webserver started at http://127.0.97.62:40145/ using document root <none> and password file <none>
I20260929 00:55:01.463907 7933 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:01.464018 7933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:01.464352 7933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:01.467236 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "80242244f2d04e058a9bb83a7f21a603"
format_stamp: "Formatted at 2026-09-29 00:55:01 on dist-test-slave-pzsx"
server_key: "ab1f106ccc20309b86b4a10866b57462"
server_key_iv: "e2dbb133f102400529a942dce7348f90"
server_key_version: "encryptionkey@0"
I20260929 00:55:01.468030 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "80242244f2d04e058a9bb83a7f21a603"
format_stamp: "Formatted at 2026-09-29 00:55:01 on dist-test-slave-pzsx"
server_key: "ab1f106ccc20309b86b4a10866b57462"
server_key_iv: "e2dbb133f102400529a942dce7348f90"
server_key_version: "encryptionkey@0"
I20260929 00:55:01.472940 7933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:55:01.476104 7948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:01.477414 7933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:01.477592 7933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "80242244f2d04e058a9bb83a7f21a603"
format_stamp: "Formatted at 2026-09-29 00:55:01 on dist-test-slave-pzsx"
server_key: "ab1f106ccc20309b86b4a10866b57462"
server_key_iv: "e2dbb133f102400529a942dce7348f90"
server_key_version: "encryptionkey@0"
I20260929 00:55:01.477799 7933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:01.497378 7933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:01.501205 7933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:01.501461 7933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:01.523203 7933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40877
I20260929 00:55:01.523224 8009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40877 every 8 connection(s)
I20260929 00:55:01.524673 7933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:01.528264 8010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:01.530997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 7933
I20260929 00:55:01.531203 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:55:01.531952 388 external_mini_cluster.cc:1428] Setting key 81353a46e60a1ab1ac9e8b224c9f5e48
I20260929 00:55:01.540428 8010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Bootstrap starting.
I20260929 00:55:01.544175 8010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:01.545511 8010 log.cc:824] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:55:01 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:01.548691 8010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: No bootstrap required, opened a new log
I20260929 00:55:01.554644 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:01.534034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.0.97.62:40877)
0929 00:55:01.534811 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:01.534822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:01.534861 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:55:01.536471 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:01.536491 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:01.536943 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:01.537263 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:01.538899 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:01.540131 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:01.541913 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:01.542583 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:01.542706 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:01.547539 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:01.547546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:01.547561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:01.547593 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:01.550696 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:01.551484 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:01.551490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:01.551495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:01.551655 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:01.552022 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:01.552025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:01.552027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:01.552582 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:01.552832 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:01.553233 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 00:55:01.553603 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":378,"threads_started":1}
I20260929 00:55:01.557320 8010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } }
I20260929 00:55:01.557678 8010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:01.557782 8010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80242244f2d04e058a9bb83a7f21a603, State: Initialized, Role: FOLLOWER
I20260929 00:55:01.558363 8010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } }
I20260929 00:55:01.558589 8010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:01.558732 8010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:01.558907 8010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:01.561038 8010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } }
I20260929 00:55:01.561503 8010 leader_election.cc:302] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80242244f2d04e058a9bb83a7f21a603; no voters:
I20260929 00:55:01.562126 8010 leader_election.cc:288] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:01.562247 8015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:01.563203 8015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [term 1 LEADER]: Becoming Leader. State: Replica: 80242244f2d04e058a9bb83a7f21a603, State: Running, Role: LEADER
I20260929 00:55:01.563706 8015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } }
I20260929 00:55:01.564400 8010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:01.567334 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80242244f2d04e058a9bb83a7f21a603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } } }
I20260929 00:55:01.567451 8017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80242244f2d04e058a9bb83a7f21a603. Latest consensus state: current_term: 1 leader_uuid: "80242244f2d04e058a9bb83a7f21a603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80242244f2d04e058a9bb83a7f21a603" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40877 } } }
I20260929 00:55:01.567808 8017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:01.567691 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:01.569108 8025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:01.574494 8025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:01.582902 8025 catalog_manager.cc:1347] Generated new cluster ID: 5d39445a65584762992b12cc4c940a1c
I20260929 00:55:01.583058 8025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:01.596122 8025 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:01.597433 8025 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:01.612574 8025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Generated new TSK 0
I20260929 00:55:01.613708 8025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 00:55:01.675992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40877
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:02.013731 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:02.013998 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:02.014101 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:02.021018 8036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:02.021133 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:02.021216 8036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:55:02.021296 8036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:02.030615 8036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:02.032226 8036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:02.034066 8036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:02.048214 8041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.049221 8042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.050032 8044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.051589 8036 server_base.cc:1034] running on GCE node
I20260929 00:55:02.052234 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:02.053134 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:02.054347 8036 hybrid_clock.cc:648] HybridClock initialized: now 1790643302054287 us; error 32 us; skew 500 ppm
I20260929 00:55:02.054639 8036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:02 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:02.057766 8036 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 00:55:02.059293 8036 webserver.cc:460] Webserver started at http://127.0.97.1:33161/ using document root <none> and password file <none>
I20260929 00:55:02.059988 8036 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:02.060115 8036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:02.060436 8036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:02.063306 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "746145d08c714ac39a5008efe23c232f"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "95e2716f2053c291bd3e2dfac65570a8"
server_key_iv: "1bcf561e32926cc89517c4cfaea31fe9"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.064086 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "746145d08c714ac39a5008efe23c232f"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "95e2716f2053c291bd3e2dfac65570a8"
server_key_iv: "1bcf561e32926cc89517c4cfaea31fe9"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.069334 8036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:55:02.072767 8051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.074033 8036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:02.074213 8036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "746145d08c714ac39a5008efe23c232f"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "95e2716f2053c291bd3e2dfac65570a8"
server_key_iv: "1bcf561e32926cc89517c4cfaea31fe9"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.074442 8036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:02.096908 8036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:02.101187 8036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:02.101469 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:02.104156 8036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:02.104274 8036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.104485 8036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:02.104624 8036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:55:02 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:02.128259 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.104595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40699 (local address 127.0.97.62:40877)
0929 00:55:02.105169 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:02.105179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:02.106404 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 00:55:02.109413 (+ 3009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:02.109418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:02.109486 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:02.109581 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:02.112156 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.113441 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.116462 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.116918 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.117047 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.120843 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:02.120849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:02.120853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:02.120893 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:02.124716 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.125562 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.125567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.125571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.125728 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.126339 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.126346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.126349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.126618 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:02.126724 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:02.127855 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 00:55:02.128013 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260929 00:55:02.130501 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.105214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.1:40699)
0929 00:55:02.106185 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.106246 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:02.109080 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:02.109913 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:02.109924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:02.110809 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:02.111873 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.111905 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.113650 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.113658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.116304 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.116315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.117208 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.117216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.117406 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.118087 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:02.118112 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:02.120590 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:02.124901 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.124912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.124934 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.125416 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.125871 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.125877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.125880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.126205 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.126778 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:02.126790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:02.127400 (+ 610us) client_negotiation.cc:769] Sending connection context
0929 00:55:02.127807 (+ 407us) client_negotiation.cc:240] Negotiation successful
0929 00:55:02.128558 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":382,"threads_started":1}
I20260929 00:55:02.151580 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.136614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47733 (local address 127.0.97.62:40877)
0929 00:55:02.136812 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:02.136818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:02.137643 (+ 825us) server_negotiation.cc:407] Connection header received
0929 00:55:02.137853 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:02.137861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:02.137965 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:02.138075 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:02.139529 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.140229 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.142121 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.142732 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.142897 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.145788 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:02.145796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:02.145801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:02.145844 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:02.148624 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.149707 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.149715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.149719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.149845 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.150283 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.150286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.150288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.150533 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:02.150647 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:02.151179 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 00:55:02.151343 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:55:02.152230 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.136682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.1:47733)
0929 00:55:02.137504 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.137528 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:02.137698 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:02.138137 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:02.138144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:02.138843 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:02.139337 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.139351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.140353 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.140360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.141885 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.141904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.142960 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.142968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.143252 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.144077 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:02.144108 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:02.145570 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:02.148770 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.148778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.148784 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.149534 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.149964 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.149972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.149978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.150167 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.150647 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:02.150655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:02.150917 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 00:55:02.151094 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 00:55:02.151559 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":443,"threads_started":1}
I20260929 00:55:02.161835 8036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35809
I20260929 00:55:02.161857 8171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35809 every 8 connection(s)
I20260929 00:55:02.163076 8036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:02.165138 8172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40877
I20260929 00:55:02.165361 8172 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:02.165972 8172 heartbeater.cc:511] Master 127.0.97.62:40877 requested a full tablet report, sending...
I20260929 00:55:02.167876 7965 ts_manager.cc:194] Registered new tserver with Master: 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809)
I20260929 00:55:02.169430 7965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40699
I20260929 00:55:02.170534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8036
I20260929 00:55:02.170759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:02.171522 388 external_mini_cluster.cc:1428] Setting key bfc85b450a79e8bb971407d0ec7f5a82
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 00:55:02.213326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40877
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:02.535190 8176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:02.535454 8176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:02.535614 8176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:02.542953 8176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:02.543057 8176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:02.543239 8176 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:55:02.543365 8176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:02.553145 8176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:02.554378 8176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:02.555693 8176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:02.568809 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.568964 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:02.569065 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:02.569559 8176 server_base.cc:1034] running on GCE node
I20260929 00:55:02.570205 8176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:02.571162 8176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:02.572360 8176 hybrid_clock.cc:648] HybridClock initialized: now 1790643302572283 us; error 57 us; skew 500 ppm
I20260929 00:55:02.572675 8176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:02 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:02.575670 8176 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 00:55:02.577046 8176 webserver.cc:460] Webserver started at http://127.0.97.2:40965/ using document root <none> and password file <none>
I20260929 00:55:02.577786 8176 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:02.577904 8176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:02.578228 8176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:02.581198 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "4a5188ac31e94d909ce8a0190a2b2c69"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "5d328b34ac8f7aa0de19dda69c2aee09"
server_key_iv: "b2fbd372c40e14aa2358f9978c762181"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.582011 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4a5188ac31e94d909ce8a0190a2b2c69"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "5d328b34ac8f7aa0de19dda69c2aee09"
server_key_iv: "b2fbd372c40e14aa2358f9978c762181"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.586860 8176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:02.590148 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.591408 8176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:02.591573 8176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "4a5188ac31e94d909ce8a0190a2b2c69"
format_stamp: "Formatted at 2026-09-29 00:55:02 on dist-test-slave-pzsx"
server_key: "5d328b34ac8f7aa0de19dda69c2aee09"
server_key_iv: "b2fbd372c40e14aa2358f9978c762181"
server_key_version: "encryptionkey@0"
I20260929 00:55:02.591787 8176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:02.606498 8176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:02.610231 8176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:02.610486 8176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:02.613093 8176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:02.613202 8176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:02.613345 8176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:02.613420 8176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:55:02 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643302, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:02.638808 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.613731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57799 (local address 127.0.97.62:40877)
0929 00:55:02.613929 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:02.613936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:02.615346 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 00:55:02.618411 (+ 3065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:02.618422 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:02.618515 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:02.618592 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:02.621546 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.622507 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.624598 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.625174 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.625315 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.630630 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:02.630638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:02.630644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:02.630686 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:02.634459 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.635417 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.635424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.635429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.635595 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.636271 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.636276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.636280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.636725 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:02.636885 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:02.638086 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0929 00:55:02.638357 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:55:02.640844 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.614305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.2:57799)
0929 00:55:02.615150 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.615214 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:02.618025 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:02.619204 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:02.619220 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:02.620144 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:02.621268 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.621297 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.622669 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.622677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.624376 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.624396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.625441 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.625454 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.625673 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.626523 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:02.626566 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:02.630360 (+ 3794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:02.634613 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.634625 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.634649 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.635266 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.635758 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.635766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.635772 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.636090 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.636965 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:02.636976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:02.637538 (+ 562us) client_negotiation.cc:769] Sending connection context
0929 00:55:02.637943 (+ 405us) client_negotiation.cc:240] Negotiation successful
0929 00:55:02.638684 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":316,"threads_started":1}
I20260929 00:55:02.658927 8176 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40503
I20260929 00:55:02.658967 8310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40503 every 8 connection(s)
I20260929 00:55:02.660184 8176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:02.662284 8311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40877
I20260929 00:55:02.662542 8311 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:02.662595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8176
I20260929 00:55:02.662791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:02.663300 388 external_mini_cluster.cc:1428] Setting key 7718a11e86a5508af433f78cb600c423
I20260929 00:55:02.663375 8311 heartbeater.cc:511] Master 127.0.97.62:40877 requested a full tablet report, sending...
I20260929 00:55:02.664817 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.651642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.2:54919)
0929 00:55:02.652329 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:02.652355 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:02.652558 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:02.652914 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:02.652919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:02.653421 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:02.653703 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.653713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.654754 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.654760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.655748 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:02.655759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.656521 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.656528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:02.656698 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.657361 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:02.657377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:02.658410 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:02.661899 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.661905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.661909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.662405 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.662812 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:02.662818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:02.662822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:02.662978 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:02.663683 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:02.663687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:02.663905 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 00:55:02.664067 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 00:55:02.664370 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":441,"threads_started":1}
I20260929 00:55:02.664858 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:02.651507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54919 (local address 127.0.97.62:40877)
0929 00:55:02.651702 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:02.651709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:02.652455 (+ 746us) server_negotiation.cc:407] Connection header received
0929 00:55:02.652703 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:02.652709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:02.652783 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:02.652886 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:02.653863 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.654597 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.655886 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:02.656317 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:02.656456 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:02.658593 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:02.658599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:02.658603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:02.658645 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:02.661767 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.662515 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.662521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.662527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.662685 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:02.663102 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:02.663106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:02.663109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:02.663539 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:02.663704 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:02.664188 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 00:55:02.664454 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:55:02.664989 7964 ts_manager.cc:194] Registered new tserver with Master: 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503)
I20260929 00:55:02.666113 7964 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57799
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 00:55:02.709546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40877
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:03.028553 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:03.028832 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:03.028927 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:03.036368 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:03.036478 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:03.036588 8315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 00:55:03.036662 8315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:03.046557 8315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:03.047922 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:03.049314 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:03.062477 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:03.062503 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:03.062633 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:03.063527 8315 server_base.cc:1034] running on GCE node
I20260929 00:55:03.064163 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:03.065099 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:03.066314 8315 hybrid_clock.cc:648] HybridClock initialized: now 1790643303066260 us; error 33 us; skew 500 ppm
I20260929 00:55:03.066570 8315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643303, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:03.069641 8315 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 00:55:03.071002 8315 webserver.cc:460] Webserver started at http://127.0.97.3:41185/ using document root <none> and password file <none>
I20260929 00:55:03.071761 8315 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:03.071848 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:03.072113 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:03.074994 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "0ab4c4752cdb44429c8d6242ef531954"
format_stamp: "Formatted at 2026-09-29 00:55:03 on dist-test-slave-pzsx"
server_key: "417572a3b1ba2312c925229abc0a8ccf"
server_key_iv: "d12bdad4918b90e7dafbf0f0f60c1a79"
server_key_version: "encryptionkey@0"
I20260929 00:55:03.075700 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0ab4c4752cdb44429c8d6242ef531954"
format_stamp: "Formatted at 2026-09-29 00:55:03 on dist-test-slave-pzsx"
server_key: "417572a3b1ba2312c925229abc0a8ccf"
server_key_iv: "d12bdad4918b90e7dafbf0f0f60c1a79"
server_key_version: "encryptionkey@0"
I20260929 00:55:03.080825 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 00:55:03.084318 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:03.085608 8315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:03.085764 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "0ab4c4752cdb44429c8d6242ef531954"
format_stamp: "Formatted at 2026-09-29 00:55:03 on dist-test-slave-pzsx"
server_key: "417572a3b1ba2312c925229abc0a8ccf"
server_key_iv: "d12bdad4918b90e7dafbf0f0f60c1a79"
server_key_version: "encryptionkey@0"
I20260929 00:55:03.085985 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:03.114904 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:03.119422 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:03.119710 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:03.122341 8315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:03.122421 8315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:03.122563 8315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:03.122646 8315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643303, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:03.150004 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.123932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.3:46297)
0929 00:55:03.124799 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.124837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.126755 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.127557 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.127572 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.128423 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:03.129478 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.129512 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.131295 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.131300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.133854 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.133866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.134917 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.134928 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.135242 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.136131 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:03.136172 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:03.140025 (+ 3853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:03.144254 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.144270 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.144293 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.145187 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.145657 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.145662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.145666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.146007 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.146763 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:03.146774 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:03.147192 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.147614 (+ 422us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.148271 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":339,"threads_started":1}
I20260929 00:55:03.151046 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.123335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46297 (local address 127.0.97.62:40877)
0929 00:55:03.123504 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.123508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.124955 (+ 1447us) server_negotiation.cc:407] Connection header received
0929 00:55:03.127030 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.127036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.127123 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.127254 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.129735 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.131070 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.134010 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.134648 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.134807 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.140354 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.140363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.140370 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.140415 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.144077 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.145353 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.145363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.145366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.145527 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.146184 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.146188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.146191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.146617 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.146825 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.150629 (+ 3804us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.150808 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:03.171265 8315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40765
I20260929 00:55:03.171277 8449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40765 every 8 connection(s)
I20260929 00:55:03.172713 8172 heartbeater.cc:503] Master 127.0.97.62:40877 was elected leader, sending a full tablet report...
I20260929 00:55:03.172822 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:03.173204 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.158234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50143 (local address 127.0.97.62:40877)
0929 00:55:03.158430 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.158436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.158572 (+ 136us) server_negotiation.cc:407] Connection header received
0929 00:55:03.158805 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.158813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.158891 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.159012 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.160651 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.161607 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.162862 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.163288 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.163428 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.165774 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.165783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.165791 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.165844 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.169831 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.170965 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.170974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.170981 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.171141 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.171645 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.171651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.171654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.172000 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.172127 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.172643 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.172865 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:55:03.173306 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.157816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.97.3:50143)
0929 00:55:03.158461 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.158480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.158648 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.159201 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.159212 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.159906 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:03.160382 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.160403 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.161732 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.161738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.162709 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.162720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.163466 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.163472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.163621 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.164244 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:03.164261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:03.165466 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:03.170021 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.170027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.170030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.170796 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.171289 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.171296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.171302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.171498 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.172125 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:03.172132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:03.172347 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.172526 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.172848 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":475,"threads_started":1}
I20260929 00:55:03.177210 8450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40877
I20260929 00:55:03.177455 8450 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:03.178056 8450 heartbeater.cc:511] Master 127.0.97.62:40877 requested a full tablet report, sending...
I20260929 00:55:03.179349 7965 ts_manager.cc:194] Registered new tserver with Master: 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.180121 7965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46297
I20260929 00:55:03.181715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8315
I20260929 00:55:03.181874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:03.182281 388 external_mini_cluster.cc:1428] Setting key 6b5f58899b900938e30f08b09620a6e5
I20260929 00:55:03.184821 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:55:03.231783 8059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:03.221989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.0.97.62:40877)
0929 00:55:03.222155 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.222159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.222427 (+ 268us) server_negotiation.cc:407] Connection header received
0929 00:55:03.222571 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.222575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.222654 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.222746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.223981 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.224740 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.225936 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.226323 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.226408 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.231566 (+ 5158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41204: BlockingRecv error: recv got EOF from 127.0.0.1:41204 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:03.247998 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.234819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.97.62:40877)
0929 00:55:03.234996 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.235003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.235291 (+ 288us) server_negotiation.cc:407] Connection header received
0929 00:55:03.235484 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.235489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.235559 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.235664 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.236801 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.237497 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.238701 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.239120 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.239254 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.242832 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.242841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.242849 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.242890 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.245468 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.246137 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.246143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.246147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.246271 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.246632 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.246637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.246640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.246912 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.247186 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.247471 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.247691 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:55:03.251659 7965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:03.258878 7965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:03.292943 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.270748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40503 (local address 127.0.0.1:41750)
0929 00:55:03.282192 (+ 11444us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.282232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.282425 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.284009 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.284019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.284054 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.284380 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.284389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.286743 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.286751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.289958 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.289974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.291991 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.291997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.292196 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.292246 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.292389 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.292586 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11150,"thread_start_us":12954,"threads_started":1}
I20260929 00:55:03.292966 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.271235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.97.2:40503)
0929 00:55:03.280814 (+ 9579us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.280827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.282624 (+ 1797us) server_negotiation.cc:407] Connection header received
0929 00:55:03.282754 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.282763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.283073 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.283323 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.284564 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.286525 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.290111 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.291528 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.291657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.291872 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.292489 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.292684 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9415,"thread_start_us":9287,"threads_started":1}
I20260929 00:55:03.296254 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.270868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35809 (local address 127.0.0.1:58306)
0929 00:55:03.283980 (+ 13112us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.283996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.284088 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.288967 (+ 4879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.288972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.289007 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.289541 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.289560 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.291463 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.291468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.293461 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.293475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.295477 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.295484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.295684 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.295716 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.295819 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.295940 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12979,"thread_start_us":10862,"threads_started":1}
I20260929 00:55:03.296327 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.271757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.97.1:35809)
0929 00:55:03.288391 (+ 16634us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.288398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.288420 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:55:03.288489 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.288540 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.288767 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.289007 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.289778 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.291314 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.293672 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.295033 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.295150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.295347 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.295885 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.296052 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16506,"thread_start_us":17322,"threads_started":1}
I20260929 00:55:03.297566 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.271241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40765 (local address 127.0.0.1:39794)
0929 00:55:03.289869 (+ 18628us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.289891 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.290001 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.290747 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.290751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.290776 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.291126 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.291138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.293096 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.293103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.295003 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.295014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.296877 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.296883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.297048 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.297077 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.297170 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.297284 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18507,"thread_start_us":18355,"threads_started":1}
I20260929 00:55:03.297812 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.271542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.0.97.3:40765)
0929 00:55:03.284442 (+ 12900us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.284454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.290118 (+ 5664us) server_negotiation.cc:407] Connection header received
0929 00:55:03.290251 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.290259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.290542 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.290751 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.291345 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.292933 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.295159 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.296686 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.296820 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.297019 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.297223 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.297378 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12679,"thread_start_us":12954,"threads_started":1}
I20260929 00:55:03.297683 8241 tablet_service.cc:1467] Processing CreateTablet for tablet 343384508ea5476c8c2995c01033d162 (DEFAULT_TABLE table=test-table [id=51d717cb12b5463e918d8dca70f7cbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.299120 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343384508ea5476c8c2995c01033d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.299696 8102 tablet_service.cc:1467] Processing CreateTablet for tablet 343384508ea5476c8c2995c01033d162 (DEFAULT_TABLE table=test-table [id=51d717cb12b5463e918d8dca70f7cbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.300967 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343384508ea5476c8c2995c01033d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.301149 8380 tablet_service.cc:1467] Processing CreateTablet for tablet 343384508ea5476c8c2995c01033d162 (DEFAULT_TABLE table=test-table [id=51d717cb12b5463e918d8dca70f7cbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.302400 8380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343384508ea5476c8c2995c01033d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.313210 8474 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Bootstrap starting.
I20260929 00:55:03.315001 8475 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Bootstrap starting.
I20260929 00:55:03.316443 8476 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Bootstrap starting.
I20260929 00:55:03.317982 8474 tablet_bootstrap.cc:654] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.318440 8475 tablet_bootstrap.cc:654] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.319298 8474 log.cc:824] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:03.319361 8476 tablet_bootstrap.cc:654] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.319588 8475 log.cc:824] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:03.320457 8476 log.cc:824] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:03.322558 8475 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: No bootstrap required, opened a new log
I20260929 00:55:03.322856 8475 ts_tablet_manager.cc:1397] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 00:55:03.323673 8474 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: No bootstrap required, opened a new log
I20260929 00:55:03.323993 8474 ts_tablet_manager.cc:1397] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 00:55:03.329496 8476 tablet_bootstrap.cc:492] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: No bootstrap required, opened a new log
I20260929 00:55:03.329767 8476 ts_tablet_manager.cc:1397] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 00:55:03.330411 8475 raft_consensus.cc:348] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.330785 8475 raft_consensus.cc:374] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.330914 8475 raft_consensus.cc:729] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.331532 8475 consensus_queue.cc:260] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.333261 8475 ts_tablet_manager.cc:1428] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260929 00:55:03.335273 8474 raft_consensus.cc:348] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.335681 8474 raft_consensus.cc:374] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.335778 8474 raft_consensus.cc:729] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.336349 8474 consensus_queue.cc:260] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.337411 8311 heartbeater.cc:503] Master 127.0.97.62:40877 was elected leader, sending a full tablet report...
I20260929 00:55:03.337986 8474 ts_tablet_manager.cc:1428] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 00:55:03.340919 8476 raft_consensus.cc:348] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.341394 8476 raft_consensus.cc:374] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.341514 8476 raft_consensus.cc:729] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4c4752cdb44429c8d6242ef531954, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.342023 8476 consensus_queue.cc:260] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.342947 8450 heartbeater.cc:503] Master 127.0.97.62:40877 was elected leader, sending a full tablet report...
I20260929 00:55:03.343379 8476 ts_tablet_manager.cc:1428] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
W20260929 00:55:03.412679 8312 tablet.cc:2369] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:55:03.417313 8173 tablet.cc:2369] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:55:03.425530 8451 tablet.cc:2369] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:55:03.521055 8481 raft_consensus.cc:482] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:03.521394 8481 raft_consensus.cc:504] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.523229 8481 leader_election.cc:288] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.531517 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.523796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36471 (local address 127.0.97.3:40765)
0929 00:55:03.523994 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.524000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.524836 (+ 836us) server_negotiation.cc:407] Connection header received
0929 00:55:03.525173 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.525179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.525290 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.525377 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.526323 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.527389 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.529770 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.530564 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.530718 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.530840 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.531145 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.531269 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:55:03.531658 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.523486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40765 (local address 127.0.97.2:36471)
0929 00:55:03.524694 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.524721 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.524970 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.525700 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.525703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.525762 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.526105 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.526114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.527535 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.527540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.529607 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.529622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.530780 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.530787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.530968 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.531013 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.531114 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.531262 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"spinlock_wait_cycles":3456,"thread_start_us":1391,"threads_started":1}
I20260929 00:55:03.531786 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.523844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47309 (local address 127.0.97.1:35809)
0929 00:55:03.524015 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.524023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.525268 (+ 1245us) server_negotiation.cc:407] Connection header received
0929 00:55:03.525401 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.525405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.525490 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.525575 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.526245 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.526970 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.529567 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.530830 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.530956 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.531052 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.531382 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.531515 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:03.532543 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.523579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35809 (local address 127.0.97.2:47309)
0929 00:55:03.525156 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.525172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.525291 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.525612 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.525616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.525648 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.526063 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.526075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.527099 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.527107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.529405 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.529419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.531013 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.531018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.531206 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.531235 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.532009 (+ 774us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.532138 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452,"mutex_wait_us":50,"thread_start_us":614,"threads_started":1}
I20260929 00:55:03.532768 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343384508ea5476c8c2995c01033d162" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954" is_pre_election: true
I20260929 00:55:03.533373 8400 raft_consensus.cc:2454] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:03.534134 8193 leader_election.cc:302] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69; no voters:
I20260929 00:55:03.534268 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343384508ea5476c8c2995c01033d162" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f" is_pre_election: true
I20260929 00:55:03.534564 8481 raft_consensus.cc:2781] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:03.534709 8481 raft_consensus.cc:482] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:03.534718 8127 raft_consensus.cc:2454] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:03.534785 8481 raft_consensus.cc:3037] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.537869 8481 raft_consensus.cc:504] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.538522 8481 leader_election.cc:288] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Requested vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.538961 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343384508ea5476c8c2995c01033d162" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954"
I20260929 00:55:03.538960 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343384508ea5476c8c2995c01033d162" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f"
I20260929 00:55:03.539153 8127 raft_consensus.cc:3037] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.539161 8400 raft_consensus.cc:3037] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.541365 8400 raft_consensus.cc:2454] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:03.541780 8193 leader_election.cc:302] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69; no voters:
I20260929 00:55:03.542088 8481 raft_consensus.cc:2781] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:03.542188 8127 raft_consensus.cc:2454] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:03.544289 8481 raft_consensus.cc:686] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Becoming Leader. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Running, Role: LEADER
I20260929 00:55:03.544878 8481 consensus_queue.cc:237] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.549280 7964 catalog_manager.cc:5495] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:03.583621 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.578872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.0.97.2:40503)
0929 00:55:03.579044 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.579049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.579065 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:55:03.579220 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.579224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.579298 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.579421 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:03.579679 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.580366 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.581749 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.582161 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.582250 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.582628 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:03.583142 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:03.583245 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.583331 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.583389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:55:03.591537 8400 raft_consensus.cc:1261] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:03.591550 8127 raft_consensus.cc:1261] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:03.592532 8485 consensus_queue.cc:1035] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:03.592886 8481 consensus_queue.cc:1035] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:03.604254 8488 mvcc.cc:205] Tried to move back new op lower bound from 7334474971502194688 to 7334474971323858944. Current Snapshot: MvccSnapshot[applied={T|T < 7334474971502194688}]
I20260929 00:55:03.616189 7964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:03.616490 7964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:03.619807 7964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Sending DeleteTablet for 3 replicas of tablet 343384508ea5476c8c2995c01033d162
I20260929 00:55:03.620869 8241 tablet_service.cc:1514] Processing DeleteTablet for tablet 343384508ea5476c8c2995c01033d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41750
I20260929 00:55:03.620970 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet 343384508ea5476c8c2995c01033d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260929 00:55:03.621683 8380 tablet_service.cc:1514] Processing DeleteTablet for tablet 343384508ea5476c8c2995c01033d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260929 00:55:03.623792 7964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:03.624886 7964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:03.634951 8240 tablet_service.cc:1467] Processing CreateTablet for tablet fa9670d639db42578ce6077dc2358f2a (DEFAULT_TABLE table=test-table [id=00ccee2d6e384b97b1762592575aacde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.635739 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9670d639db42578ce6077dc2358f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.638520 8498 tablet_replica.cc:330] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: stopping tablet replica
I20260929 00:55:03.638970 8498 raft_consensus.cc:2229] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:03.639473 8498 raft_consensus.cc:2258] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:03.641673 8379 tablet_service.cc:1467] Processing CreateTablet for tablet fa9670d639db42578ce6077dc2358f2a (DEFAULT_TABLE table=test-table [id=00ccee2d6e384b97b1762592575aacde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.642365 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9670d639db42578ce6077dc2358f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.647750 8499 tablet_replica.cc:330] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: stopping tablet replica
I20260929 00:55:03.648257 8499 raft_consensus.cc:2229] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:03.648789 8499 raft_consensus.cc:2258] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:03.648783 8498 ts_tablet_manager.cc:1905] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:03.649710 8497 tablet_replica.cc:330] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: stopping tablet replica
I20260929 00:55:03.650098 8497 raft_consensus.cc:2229] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:03.650708 8497 raft_consensus.cc:2258] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:03.653671 8499 ts_tablet_manager.cc:1905] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:03.655719 8498 ts_tablet_manager.cc:1918] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:03.655932 8498 log.cc:1196] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/343384508ea5476c8c2995c01033d162
I20260929 00:55:03.656526 8498 ts_tablet_manager.cc:1939] T 343384508ea5476c8c2995c01033d162 P 746145d08c714ac39a5008efe23c232f: Deleting consensus metadata
I20260929 00:55:03.657155 8476 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Bootstrap starting.
I20260929 00:55:03.657464 8102 tablet_service.cc:1467] Processing CreateTablet for tablet fa9670d639db42578ce6077dc2358f2a (DEFAULT_TABLE table=test-table [id=00ccee2d6e384b97b1762592575aacde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:03.657874 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9670d639db42578ce6077dc2358f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.658856 8476 tablet_bootstrap.cc:654] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.658878 7949 catalog_manager.cc:4841] TS 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809): tablet 343384508ea5476c8c2995c01033d162 (table test-table [id=51d717cb12b5463e918d8dca70f7cbe1]) successfully deleted
I20260929 00:55:03.662257 8499 ts_tablet_manager.cc:1918] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:03.662416 8499 log.cc:1196] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/343384508ea5476c8c2995c01033d162
I20260929 00:55:03.662950 8499 ts_tablet_manager.cc:1939] T 343384508ea5476c8c2995c01033d162 P 0ab4c4752cdb44429c8d6242ef531954: Deleting consensus metadata
I20260929 00:55:03.663920 8475 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Bootstrap starting.
I20260929 00:55:03.664690 7950 catalog_manager.cc:4841] TS 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765): tablet 343384508ea5476c8c2995c01033d162 (table test-table [id=51d717cb12b5463e918d8dca70f7cbe1]) successfully deleted
I20260929 00:55:03.665539 8475 tablet_bootstrap.cc:654] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.670420 8474 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Bootstrap starting.
I20260929 00:55:03.672818 8474 tablet_bootstrap.cc:654] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.672945 8497 ts_tablet_manager.cc:1905] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:03.676429 8476 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: No bootstrap required, opened a new log
I20260929 00:55:03.676676 8475 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: No bootstrap required, opened a new log
I20260929 00:55:03.676857 8475 ts_tablet_manager.cc:1397] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260929 00:55:03.677169 8476 ts_tablet_manager.cc:1397] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20260929 00:55:03.677750 8475 raft_consensus.cc:348] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.677980 8475 raft_consensus.cc:374] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.678088 8475 raft_consensus.cc:729] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.678012 8476 raft_consensus.cc:348] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.678244 8476 raft_consensus.cc:374] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.678331 8476 raft_consensus.cc:729] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4c4752cdb44429c8d6242ef531954, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.678354 8475 consensus_queue.cc:260] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.678588 8476 consensus_queue.cc:260] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.678973 8475 ts_tablet_manager.cc:1428] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 00:55:03.679141 8476 ts_tablet_manager.cc:1428] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:55:03.681967 8474 tablet_bootstrap.cc:492] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: No bootstrap required, opened a new log
I20260929 00:55:03.682106 8474 ts_tablet_manager.cc:1397] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260929 00:55:03.682847 8474 raft_consensus.cc:348] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.683054 8474 raft_consensus.cc:374] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.683151 8474 raft_consensus.cc:729] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.683396 8474 consensus_queue.cc:260] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.683904 8474 ts_tablet_manager.cc:1428] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 00:55:03.689249 8497 ts_tablet_manager.cc:1918] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:03.689435 8497 log.cc:1196] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/343384508ea5476c8c2995c01033d162
I20260929 00:55:03.690004 8497 ts_tablet_manager.cc:1939] T 343384508ea5476c8c2995c01033d162 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting consensus metadata
I20260929 00:55:03.691711 7951 catalog_manager.cc:4841] TS 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503): tablet 343384508ea5476c8c2995c01033d162 (table test-table [id=51d717cb12b5463e918d8dca70f7cbe1]) successfully deleted
I20260929 00:55:03.791450 8481 raft_consensus.cc:482] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:03.791709 8481 raft_consensus.cc:504] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.792461 8481 leader_election.cc:288] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.792994 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9670d639db42578ce6077dc2358f2a" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954" is_pre_election: true
I20260929 00:55:03.793022 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9670d639db42578ce6077dc2358f2a" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f" is_pre_election: true
I20260929 00:55:03.793241 8400 raft_consensus.cc:2454] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:03.793268 8127 raft_consensus.cc:2454] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:03.793733 8192 leader_election.cc:302] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:03.794109 8481 raft_consensus.cc:2781] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:03.794238 8481 raft_consensus.cc:482] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:03.794353 8481 raft_consensus.cc:3037] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.796890 8481 raft_consensus.cc:504] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.797425 8481 leader_election.cc:288] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Requested vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.797894 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9670d639db42578ce6077dc2358f2a" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954"
I20260929 00:55:03.798086 8400 raft_consensus.cc:3037] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.799969 8400 raft_consensus.cc:2454] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:03.800393 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9670d639db42578ce6077dc2358f2a" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f"
I20260929 00:55:03.800410 8193 leader_election.cc:302] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69; no voters:
I20260929 00:55:03.800618 8127 raft_consensus.cc:3037] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.800706 8481 raft_consensus.cc:2781] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:03.800848 8481 raft_consensus.cc:686] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Becoming Leader. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Running, Role: LEADER
I20260929 00:55:03.801177 8481 consensus_queue.cc:237] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.803352 8127 raft_consensus.cc:2454] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:03.803869 7963 catalog_manager.cc:5495] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
Sep 29 00:55:03 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643301, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:03.846231 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.830921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.97.62:40877)
0929 00:55:03.831602 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.831612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.831856 (+ 244us) server_negotiation.cc:407] Connection header received
0929 00:55:03.832059 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.832063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.832114 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.832229 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:03.833684 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.834487 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.836266 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.836721 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.836810 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.840779 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:03.840784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:03.840787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:03.840813 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:03.843473 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.844300 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.844304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.844306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.844414 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:03.844829 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:03.844832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:03.844834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:03.845070 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:03.845189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.845644 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.845956 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":511,"thread_start_us":329,"threads_started":1}
I20260929 00:55:03.846309 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.830956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40877 (local address 127.0.0.1:41234)
0929 00:55:03.831759 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.831774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.831933 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.832305 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.832311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.833194 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:03.833483 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.833496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.834640 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.834645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.836126 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.836141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.836886 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.836892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.837077 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.837794 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:03.837818 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:03.840610 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:03.843590 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.843601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.843610 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.844190 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.844539 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:03.844544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:03.844549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:03.844724 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:03.845198 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:03.845206 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:03.845390 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.845566 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.845855 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":544,"threads_started":1}
I20260929 00:55:03.851547 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 00:55:03.852617 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:03.859333 8240 tablet_service.cc:1467] Processing CreateTablet for tablet beab9cfb3c904c7cb4ee609b65effe87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a5f22f0b8345c19907a0e4ced75d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:55:03.859320 8102 tablet_service.cc:1467] Processing CreateTablet for tablet beab9cfb3c904c7cb4ee609b65effe87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a5f22f0b8345c19907a0e4ced75d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:55:03.859627 8379 tablet_service.cc:1467] Processing CreateTablet for tablet beab9cfb3c904c7cb4ee609b65effe87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a5f22f0b8345c19907a0e4ced75d5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 00:55:03.859851 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beab9cfb3c904c7cb4ee609b65effe87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.859869 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beab9cfb3c904c7cb4ee609b65effe87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.860146 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beab9cfb3c904c7cb4ee609b65effe87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:03.865906 8476 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: Bootstrap starting.
I20260929 00:55:03.866798 8474 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: Bootstrap starting.
I20260929 00:55:03.866897 8475 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: Bootstrap starting.
I20260929 00:55:03.867345 8476 tablet_bootstrap.cc:654] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.868577 8475 tablet_bootstrap.cc:654] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.868623 8474 tablet_bootstrap.cc:654] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:03.879259 8476 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: No bootstrap required, opened a new log
I20260929 00:55:03.879462 8476 ts_tablet_manager.cc:1397] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20260929 00:55:03.880276 8476 raft_consensus.cc:348] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.880497 8476 raft_consensus.cc:374] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.880592 8476 raft_consensus.cc:729] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4c4752cdb44429c8d6242ef531954, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.880834 8475 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: No bootstrap required, opened a new log
I20260929 00:55:03.881031 8475 ts_tablet_manager.cc:1397] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260929 00:55:03.880894 8476 consensus_queue.cc:260] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.881511 8476 ts_tablet_manager.cc:1428] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 00:55:03.881408 8482 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.881765 8482 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.882566 8475 raft_consensus.cc:348] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.882754 8474 tablet_bootstrap.cc:492] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: No bootstrap required, opened a new log
I20260929 00:55:03.882899 8475 raft_consensus.cc:374] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.882937 8474 ts_tablet_manager.cc:1397] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260929 00:55:03.883050 8475 raft_consensus.cc:729] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.883363 8475 consensus_queue.cc:260] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.883796 8474 raft_consensus.cc:348] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.884048 8474 raft_consensus.cc:374] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:03.883854 8480 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.884133 8474 raft_consensus.cc:729] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Initialized, Role: FOLLOWER
I20260929 00:55:03.884169 8480 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.884390 8474 consensus_queue.cc:260] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.885129 8481 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.885471 8481 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.886391 8474 ts_tablet_manager.cc:1428] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 00:55:03.887739 8475 ts_tablet_manager.cc:1428] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 00:55:03.907193 8480 raft_consensus.cc:482] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:03.907431 8480 raft_consensus.cc:504] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.909338 8480 leader_election.cc:288] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.917517 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.910335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47753 (local address 127.0.97.3:40765)
0929 00:55:03.910545 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.910553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.910576 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:55:03.910664 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.910672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.910767 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.910855 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.911514 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.912649 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.915185 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.916471 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.916637 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.916744 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.917092 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.917238 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:55:03.917579 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.909527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40765 (local address 127.0.97.1:47753)
0929 00:55:03.910290 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.910312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.910442 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.910892 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.910896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.910928 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.911324 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.911336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.912775 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.912783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.915002 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.915017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.916712 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.916717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.916914 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.916949 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.917047 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.917196 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":4,"thread_start_us":463,"threads_started":1}
I20260929 00:55:03.917621 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.909767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40839 (local address 127.0.97.2:40503)
0929 00:55:03.910173 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.910181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.911054 (+ 873us) server_negotiation.cc:407] Connection header received
0929 00:55:03.911233 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.911238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.911327 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.911444 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:03.912060 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.913266 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.915571 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.916415 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.916632 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.916735 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.917205 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.917341 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260929 00:55:03.918309 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.909527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40503 (local address 127.0.97.1:40839)
0929 00:55:03.910930 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.910948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.911091 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.911460 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.911466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.911492 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:03.911865 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.911878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.913399 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.913403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.915430 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.915439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.916864 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.916870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.917037 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.917066 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.917776 (+ 710us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.917928 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":255,"thread_start_us":887,"threads_started":1}
I20260929 00:55:03.918453 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beab9cfb3c904c7cb4ee609b65effe87" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954" is_pre_election: true
I20260929 00:55:03.918735 8400 raft_consensus.cc:2454] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 0.
I20260929 00:55:03.919201 8053 leader_election.cc:302] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:03.919310 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beab9cfb3c904c7cb4ee609b65effe87" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" is_pre_election: true
I20260929 00:55:03.919602 8480 raft_consensus.cc:2781] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:03.919726 8480 raft_consensus.cc:482] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:03.919811 8480 raft_consensus.cc:3037] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.919770 8261 raft_consensus.cc:2454] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 0.
I20260929 00:55:03.922997 8480 raft_consensus.cc:504] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.923725 8480 leader_election.cc:288] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 election: Requested vote from peers 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:03.924185 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beab9cfb3c904c7cb4ee609b65effe87" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954"
I20260929 00:55:03.924168 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beab9cfb3c904c7cb4ee609b65effe87" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5188ac31e94d909ce8a0190a2b2c69"
I20260929 00:55:03.924412 8261 raft_consensus.cc:3037] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.924422 8400 raft_consensus.cc:3037] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:03.927052 8261 raft_consensus.cc:2454] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 1.
I20260929 00:55:03.927223 8400 raft_consensus.cc:2454] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 1.
I20260929 00:55:03.927443 8054 leader_election.cc:302] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:03.927779 8480 raft_consensus.cc:2781] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:03.928279 8480 raft_consensus.cc:686] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [term 1 LEADER]: Becoming Leader. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Running, Role: LEADER
I20260929 00:55:03.928706 8480 consensus_queue.cc:237] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:03.930531 8516 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: TxnStatusTablet state changed. Reason: New leader 746145d08c714ac39a5008efe23c232f. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.930932 8516 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:55:03.932525 7964 catalog_manager.cc:5495] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f reported cstate change: term changed from 0 to 1, leader changed from <none> to 746145d08c714ac39a5008efe23c232f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:03.940903 8518 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 00:55:03.941145 8518 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 00:55:03.957042 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.949572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.0.97.1:35809)
0929 00:55:03.949766 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:03.949772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:03.950087 (+ 315us) server_negotiation.cc:407] Connection header received
0929 00:55:03.950353 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:03.950359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:03.950487 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:03.950640 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:03.951052 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.952266 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.954142 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.954846 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.954989 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.955570 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:03.956295 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:03.956441 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:03.956602 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 00:55:03.956697 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:55:03.957059 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:03.949609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35809 (local address 127.0.0.1:58318)
0929 00:55:03.949974 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:03.949994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:03.950139 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:03.950683 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:03.950689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:03.950703 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:55:03.950890 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.950900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.952417 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.952422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.953981 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:03.953998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:03.955013 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:03.955019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:03.955274 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:03.955342 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:55:03.956422 (+ 1080us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:55:03.956443 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:55:03.956569 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 00:55:03.956738 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260929 00:55:03.961625 8400 raft_consensus.cc:1261] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Refusing update from remote peer 746145d08c714ac39a5008efe23c232f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:03.961848 8261 raft_consensus.cc:1261] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Refusing update from remote peer 746145d08c714ac39a5008efe23c232f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:03.962596 8516 consensus_queue.cc:1035] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:03.962999 8480 consensus_queue.cc:1035] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:03.978288 8482 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: TxnStatusTablet state changed. Reason: New leader 746145d08c714ac39a5008efe23c232f. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.978246 8481 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: TxnStatusTablet state changed. Reason: New leader 746145d08c714ac39a5008efe23c232f. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.978549 8482 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.978612 8481 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.981098 8480 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.981398 8480 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:55:03.984807 8482 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.985038 8482 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 0ab4c4752cdb44429c8d6242ef531954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:03.991741 8516 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.992027 8516 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 746145d08c714ac39a5008efe23c232f: This TxnStatusTablet replica's current role is: LEADER
I20260929 00:55:03.994460 8481 tablet_replica.cc:439] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } } }
I20260929 00:55:03.994788 8481 tablet_replica.cc:442] T beab9cfb3c904c7cb4ee609b65effe87 P 4a5188ac31e94d909ce8a0190a2b2c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 00:55:04.015985 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51125 (local address 127.0.97.1:35809)
0929 00:55:04.007684 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.007691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.008709 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 00:55:04.009063 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.009069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.009185 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.009275 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:04.010065 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.011222 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.013696 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.014399 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.014561 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.015124 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.015281 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.015383 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.015559 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.015653 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:55:04.016085 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.007485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35809 (local address 127.0.97.2:51125)
0929 00:55:04.008603 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:04.008625 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:04.008850 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:04.009489 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:04.009494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:04.009507 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:55:04.009846 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:04.009858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.011445 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.011450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:04.013518 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:04.013539 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.014638 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.014645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:04.014847 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.014922 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.015398 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.015413 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 00:55:04.015511 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 00:55:04.015671 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":731,"threads_started":1}
I20260929 00:55:04.029067 8399 raft_consensus.cc:1261] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.029654 8127 raft_consensus.cc:1261] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.030297 8481 consensus_queue.cc:1035] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.030753 8485 consensus_queue.cc:1035] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.085180 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.076266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50073 (local address 127.0.97.2:40503)
0929 00:55:04.076424 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.076432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.076982 (+ 550us) server_negotiation.cc:407] Connection header received
0929 00:55:04.077247 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.077253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.077355 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.077415 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:04.078189 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.079380 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.081660 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.082472 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.082623 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.083157 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.083520 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.084562 (+ 1042us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.084779 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.084878 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 00:55:04.086582 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.075885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40503 (local address 127.0.97.1:50073)
0929 00:55:04.076884 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:04.076908 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:04.077093 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:04.077607 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:04.077612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:04.077624 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 00:55:04.078024 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:04.078041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.079551 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.079556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:04.081494 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:04.081513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.082667 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.082674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:04.082862 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.082940 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.084620 (+ 1680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.084635 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 00:55:04.085813 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0929 00:55:04.086140 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":1465,"threads_started":1}
I20260929 00:55:04.140799 7965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.141088 7965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.144542 7965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Sending DeleteTablet for 3 replicas of tablet fa9670d639db42578ce6077dc2358f2a
I20260929 00:55:04.145468 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9670d639db42578ce6077dc2358f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41750
I20260929 00:55:04.145735 8497 tablet_replica.cc:330] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: stopping tablet replica
I20260929 00:55:04.145938 8497 raft_consensus.cc:2229] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:04.146108 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9670d639db42578ce6077dc2358f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260929 00:55:04.146274 8497 raft_consensus.cc:2258] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.146551 8498 tablet_replica.cc:330] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: stopping tablet replica
I20260929 00:55:04.146736 8498 raft_consensus.cc:2229] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.146736 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9670d639db42578ce6077dc2358f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260929 00:55:04.146935 8498 raft_consensus.cc:2258] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.147368 8499 tablet_replica.cc:330] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: stopping tablet replica
I20260929 00:55:04.147579 8499 raft_consensus.cc:2229] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.147768 8499 raft_consensus.cc:2258] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.148232 8498 ts_tablet_manager.cc:1905] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.148842 8497 ts_tablet_manager.cc:1905] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.153453 8499 ts_tablet_manager.cc:1905] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.160261 8499 ts_tablet_manager.cc:1918] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.160491 8499 log.cc:1196] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/fa9670d639db42578ce6077dc2358f2a
I20260929 00:55:04.161059 8499 ts_tablet_manager.cc:1939] T fa9670d639db42578ce6077dc2358f2a P 0ab4c4752cdb44429c8d6242ef531954: Deleting consensus metadata
I20260929 00:55:04.165556 8497 ts_tablet_manager.cc:1918] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.165799 8497 log.cc:1196] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/fa9670d639db42578ce6077dc2358f2a
I20260929 00:55:04.166273 8497 ts_tablet_manager.cc:1939] T fa9670d639db42578ce6077dc2358f2a P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting consensus metadata
I20260929 00:55:04.167270 8498 ts_tablet_manager.cc:1918] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.167425 8498 log.cc:1196] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/fa9670d639db42578ce6077dc2358f2a
I20260929 00:55:04.167485 7951 catalog_manager.cc:4841] TS 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503): tablet fa9670d639db42578ce6077dc2358f2a (table test-table [id=00ccee2d6e384b97b1762592575aacde]) successfully deleted
I20260929 00:55:04.167842 8498 ts_tablet_manager.cc:1939] T fa9670d639db42578ce6077dc2358f2a P 746145d08c714ac39a5008efe23c232f: Deleting consensus metadata
I20260929 00:55:04.168921 7949 catalog_manager.cc:4841] TS 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809): tablet fa9670d639db42578ce6077dc2358f2a (table test-table [id=00ccee2d6e384b97b1762592575aacde]) successfully deleted
I20260929 00:55:04.177628 7950 catalog_manager.cc:4841] TS 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765): tablet fa9670d639db42578ce6077dc2358f2a (table test-table [id=00ccee2d6e384b97b1762592575aacde]) successfully deleted
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:55:04.239333 8509 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:04.224036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.97.62:40877)
0929 00:55:04.224219 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.224223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.224743 (+ 520us) server_negotiation.cc:407] Connection header received
0929 00:55:04.224870 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.224875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.224962 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.225084 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:04.226388 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.227497 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.228819 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.229487 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.229611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.239069 (+ 9458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41240: BlockingRecv error: recv got EOF from 127.0.0.1:41240 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 00:55:04.256986 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.242813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.97.62:40877)
0929 00:55:04.243027 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.243034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.243323 (+ 289us) server_negotiation.cc:407] Connection header received
0929 00:55:04.243530 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.243536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.243615 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.243738 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:04.245071 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.245806 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.247004 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.247431 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.247560 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.251424 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:04.251429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:04.251432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:04.251465 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:04.254369 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.255000 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.255007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.255011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.255139 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.255539 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.255546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.255549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.255820 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:04.256085 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.256379 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.256640 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 00:55:04.260705 7965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:04.261312 7965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:04.268446 8102 tablet_service.cc:1467] Processing CreateTablet for tablet 282ad9ebbc734392a77974e575f8c324 (DEFAULT_TABLE table=test-table [id=359467f767fb47aa80ff062040ebff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.268532 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 282ad9ebbc734392a77974e575f8c324 (DEFAULT_TABLE table=test-table [id=359467f767fb47aa80ff062040ebff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.268965 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282ad9ebbc734392a77974e575f8c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.269011 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282ad9ebbc734392a77974e575f8c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.274757 8475 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Bootstrap starting.
I20260929 00:55:04.275290 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 282ad9ebbc734392a77974e575f8c324 (DEFAULT_TABLE table=test-table [id=359467f767fb47aa80ff062040ebff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.275731 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282ad9ebbc734392a77974e575f8c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.276161 8475 tablet_bootstrap.cc:654] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.278616 8475 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: No bootstrap required, opened a new log
I20260929 00:55:04.278709 8474 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Bootstrap starting.
I20260929 00:55:04.278852 8475 ts_tablet_manager.cc:1397] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 00:55:04.279680 8475 raft_consensus.cc:348] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.279976 8475 raft_consensus.cc:374] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.280073 8475 raft_consensus.cc:729] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.280329 8474 tablet_bootstrap.cc:654] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.280328 8475 consensus_queue.cc:260] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.281031 8475 ts_tablet_manager.cc:1428] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:04.283000 8476 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Bootstrap starting.
I20260929 00:55:04.284811 8474 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: No bootstrap required, opened a new log
I20260929 00:55:04.284951 8474 ts_tablet_manager.cc:1397] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 00:55:04.285156 8476 tablet_bootstrap.cc:654] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.285734 8474 raft_consensus.cc:348] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.285975 8474 raft_consensus.cc:374] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.286077 8474 raft_consensus.cc:729] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.286339 8474 consensus_queue.cc:260] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.286907 8474 ts_tablet_manager.cc:1428] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 00:55:04.287941 8476 tablet_bootstrap.cc:492] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: No bootstrap required, opened a new log
I20260929 00:55:04.288105 8476 ts_tablet_manager.cc:1397] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 00:55:04.288946 8476 raft_consensus.cc:348] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.289258 8476 raft_consensus.cc:374] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.289359 8476 raft_consensus.cc:729] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4c4752cdb44429c8d6242ef531954, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.289580 8476 consensus_queue.cc:260] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.290153 8476 ts_tablet_manager.cc:1428] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:04.466600 8526 raft_consensus.cc:482] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:04.466840 8526 raft_consensus.cc:504] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.467506 8526 leader_election.cc:288] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:04.468084 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282ad9ebbc734392a77974e575f8c324" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" is_pre_election: true
I20260929 00:55:04.468065 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282ad9ebbc734392a77974e575f8c324" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954" is_pre_election: true
I20260929 00:55:04.468319 8261 raft_consensus.cc:2454] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 0.
I20260929 00:55:04.468339 8400 raft_consensus.cc:2454] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 0.
I20260929 00:55:04.468799 8053 leader_election.cc:302] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:04.469075 8526 raft_consensus.cc:2781] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:04.469174 8526 raft_consensus.cc:482] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:04.469303 8526 raft_consensus.cc:3037] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.471325 8526 raft_consensus.cc:504] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.471840 8526 leader_election.cc:288] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 election: Requested vote from peers 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:04.472344 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282ad9ebbc734392a77974e575f8c324" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5188ac31e94d909ce8a0190a2b2c69"
I20260929 00:55:04.472325 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282ad9ebbc734392a77974e575f8c324" candidate_uuid: "746145d08c714ac39a5008efe23c232f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954"
I20260929 00:55:04.472558 8261 raft_consensus.cc:3037] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.472575 8400 raft_consensus.cc:3037] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.475402 8400 raft_consensus.cc:2454] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 1.
I20260929 00:55:04.475417 8261 raft_consensus.cc:2454] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746145d08c714ac39a5008efe23c232f in term 1.
I20260929 00:55:04.475811 8054 leader_election.cc:302] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:04.476118 8526 raft_consensus.cc:2781] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:04.476261 8526 raft_consensus.cc:686] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 1 LEADER]: Becoming Leader. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Running, Role: LEADER
I20260929 00:55:04.476580 8526 consensus_queue.cc:237] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.479069 7964 catalog_manager.cc:5495] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f reported cstate change: term changed from 0 to 1, leader changed from <none> to 746145d08c714ac39a5008efe23c232f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "746145d08c714ac39a5008efe23c232f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:04.532797 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.527536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.97.1:35809)
0929 00:55:04.527962 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.527968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.527988 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:04.528052 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.528056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.528148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.528275 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:04.528570 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.529605 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.530870 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.531507 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.531643 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.532009 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.532130 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.532241 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.532387 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.532461 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":199,"threads_started":1}
I20260929 00:55:04.535692 8400 raft_consensus.cc:1261] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Refusing update from remote peer 746145d08c714ac39a5008efe23c232f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.536275 8261 raft_consensus.cc:1261] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Refusing update from remote peer 746145d08c714ac39a5008efe23c232f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.536305 8516 consensus_queue.cc:1035] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.536804 8516 consensus_queue.cc:1035] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.540275 8489 mvcc.cc:205] Tried to move back new op lower bound from 7334474975373000704 to 7334474975138279424. Current Snapshot: MvccSnapshot[applied={T|T < 7334474975373000704}]
I20260929 00:55:04.542333 8490 mvcc.cc:205] Tried to move back new op lower bound from 7334474975373000704 to 7334474975138279424. Current Snapshot: MvccSnapshot[applied={T|T < 7334474975373000704 or (T in {7334474975373000704})}]
I20260929 00:55:04.551374 7963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.551604 7963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.554605 7963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Sending DeleteTablet for 3 replicas of tablet 282ad9ebbc734392a77974e575f8c324
I20260929 00:55:04.555383 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet 282ad9ebbc734392a77974e575f8c324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41750
I20260929 00:55:04.555418 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet 282ad9ebbc734392a77974e575f8c324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260929 00:55:04.555650 8497 tablet_replica.cc:330] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: stopping tablet replica
I20260929 00:55:04.555593 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 282ad9ebbc734392a77974e575f8c324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260929 00:55:04.555670 8498 tablet_replica.cc:330] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: stopping tablet replica
I20260929 00:55:04.555815 8497 raft_consensus.cc:2229] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.555836 8498 raft_consensus.cc:2229] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:04.556044 8497 raft_consensus.cc:2258] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.556327 8498 raft_consensus.cc:2258] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.556659 8499 tablet_replica.cc:330] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: stopping tablet replica
I20260929 00:55:04.556860 8499 raft_consensus.cc:2229] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.557103 8499 raft_consensus.cc:2258] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.557188 8497 ts_tablet_manager.cc:1905] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.558480 8499 ts_tablet_manager.cc:1905] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.558588 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:04.559151 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:04.562112 8498 ts_tablet_manager.cc:1905] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.564779 8499 ts_tablet_manager.cc:1918] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:04.564983 8499 log.cc:1196] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/282ad9ebbc734392a77974e575f8c324
I20260929 00:55:04.565829 8102 tablet_service.cc:1467] Processing CreateTablet for tablet 5568e883283d4f6797b4b933497df9b3 (DEFAULT_TABLE table=test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.565969 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 5568e883283d4f6797b4b933497df9b3 (DEFAULT_TABLE table=test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.566259 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5568e883283d4f6797b4b933497df9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.566346 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5568e883283d4f6797b4b933497df9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.569193 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 5568e883283d4f6797b4b933497df9b3 (DEFAULT_TABLE table=test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:04.571046 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5568e883283d4f6797b4b933497df9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:04.571184 8499 ts_tablet_manager.cc:1939] T 282ad9ebbc734392a77974e575f8c324 P 0ab4c4752cdb44429c8d6242ef531954: Deleting consensus metadata
I20260929 00:55:04.571328 8475 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Bootstrap starting.
I20260929 00:55:04.572995 8475 tablet_bootstrap.cc:654] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.576778 7950 catalog_manager.cc:4841] TS 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765): tablet 282ad9ebbc734392a77974e575f8c324 (table test-table [id=359467f767fb47aa80ff062040ebff4b]) successfully deleted
I20260929 00:55:04.577780 8476 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Bootstrap starting.
I20260929 00:55:04.579372 8476 tablet_bootstrap.cc:654] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.584219 8497 ts_tablet_manager.cc:1918] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:04.584407 8497 log.cc:1196] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/282ad9ebbc734392a77974e575f8c324
I20260929 00:55:04.584939 8497 ts_tablet_manager.cc:1939] T 282ad9ebbc734392a77974e575f8c324 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting consensus metadata
I20260929 00:55:04.588176 8474 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Bootstrap starting.
I20260929 00:55:04.590410 8474 tablet_bootstrap.cc:654] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:04.592687 8474 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: No bootstrap required, opened a new log
I20260929 00:55:04.592862 8474 ts_tablet_manager.cc:1397] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 00:55:04.593541 8474 raft_consensus.cc:348] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.593824 8474 raft_consensus.cc:374] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.593914 8474 raft_consensus.cc:729] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.593940 8476 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: No bootstrap required, opened a new log
I20260929 00:55:04.594035 8498 ts_tablet_manager.cc:1918] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:04.594168 8498 log.cc:1196] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/282ad9ebbc734392a77974e575f8c324
I20260929 00:55:04.594312 8476 ts_tablet_manager.cc:1397] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260929 00:55:04.594158 8474 consensus_queue.cc:260] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.595094 8476 raft_consensus.cc:348] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.595660 8474 ts_tablet_manager.cc:1428] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:04.595897 8498 ts_tablet_manager.cc:1939] T 282ad9ebbc734392a77974e575f8c324 P 746145d08c714ac39a5008efe23c232f: Deleting consensus metadata
I20260929 00:55:04.596691 7951 catalog_manager.cc:4841] TS 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503): tablet 282ad9ebbc734392a77974e575f8c324 (table test-table [id=359467f767fb47aa80ff062040ebff4b]) successfully deleted
I20260929 00:55:04.597350 8476 raft_consensus.cc:374] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.597450 8476 raft_consensus.cc:729] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4c4752cdb44429c8d6242ef531954, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.597450 7949 catalog_manager.cc:4841] TS 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809): tablet 282ad9ebbc734392a77974e575f8c324 (table test-table [id=359467f767fb47aa80ff062040ebff4b]) successfully deleted
I20260929 00:55:04.597723 8476 consensus_queue.cc:260] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.598359 8476 ts_tablet_manager.cc:1428] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 00:55:04.599530 8475 tablet_bootstrap.cc:492] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: No bootstrap required, opened a new log
I20260929 00:55:04.599715 8475 ts_tablet_manager.cc:1397] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.015s
I20260929 00:55:04.600559 8475 raft_consensus.cc:348] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.600814 8475 raft_consensus.cc:374] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:04.600939 8475 raft_consensus.cc:729] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746145d08c714ac39a5008efe23c232f, State: Initialized, Role: FOLLOWER
I20260929 00:55:04.601186 8475 consensus_queue.cc:260] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.601769 8475 ts_tablet_manager.cc:1428] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 00:55:04.609684 8485 raft_consensus.cc:482] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:04.609874 8485 raft_consensus.cc:504] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.610450 8485 leader_election.cc:288] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:04.610839 8126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5568e883283d4f6797b4b933497df9b3" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f" is_pre_election: true
I20260929 00:55:04.610880 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5568e883283d4f6797b4b933497df9b3" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954" is_pre_election: true
I20260929 00:55:04.611085 8126 raft_consensus.cc:2454] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:04.611119 8400 raft_consensus.cc:2454] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 0.
I20260929 00:55:04.611588 8193 leader_election.cc:302] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ab4c4752cdb44429c8d6242ef531954, 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:04.611886 8485 raft_consensus.cc:2781] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:04.612013 8485 raft_consensus.cc:482] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:04.612123 8485 raft_consensus.cc:3037] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.614413 8485 raft_consensus.cc:504] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.615056 8485 leader_election.cc:288] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Requested vote from peers 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809), 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765)
I20260929 00:55:04.615509 8126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5568e883283d4f6797b4b933497df9b3" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746145d08c714ac39a5008efe23c232f"
I20260929 00:55:04.615510 8400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5568e883283d4f6797b4b933497df9b3" candidate_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab4c4752cdb44429c8d6242ef531954"
I20260929 00:55:04.615696 8126 raft_consensus.cc:3037] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.615718 8400 raft_consensus.cc:3037] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:04.618183 8126 raft_consensus.cc:2454] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:04.618605 8400 raft_consensus.cc:2454] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5188ac31e94d909ce8a0190a2b2c69 in term 1.
I20260929 00:55:04.618671 8192 leader_election.cc:302] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5188ac31e94d909ce8a0190a2b2c69, 746145d08c714ac39a5008efe23c232f; no voters:
I20260929 00:55:04.618964 8485 raft_consensus.cc:2781] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:04.619369 8485 raft_consensus.cc:686] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Becoming Leader. State: Replica: 4a5188ac31e94d909ce8a0190a2b2c69, State: Running, Role: LEADER
I20260929 00:55:04.619690 8485 consensus_queue.cc:237] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } }
I20260929 00:55:04.621889 7965 catalog_manager.cc:5495] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a5188ac31e94d909ce8a0190a2b2c69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:04.650563 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.644870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.0.97.2:40503)
0929 00:55:04.645680 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.645686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.645703 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:55:04.645758 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.645764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.645839 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.645941 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:04.646297 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.647417 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.648697 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.649353 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.649474 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.649864 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:04.649981 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:04.650070 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.650171 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.650241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":559,"threads_started":1}
I20260929 00:55:04.659696 8400 raft_consensus.cc:1261] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.659763 8126 raft_consensus.cc:1261] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Refusing update from remote peer 4a5188ac31e94d909ce8a0190a2b2c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:04.661068 8485 consensus_queue.cc:1035] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab4c4752cdb44429c8d6242ef531954" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.661476 8556 consensus_queue.cc:1035] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746145d08c714ac39a5008efe23c232f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:04.737567 7965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.737849 7965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:04.740681 7965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80242244f2d04e058a9bb83a7f21a603: Sending DeleteTablet for 3 replicas of tablet 5568e883283d4f6797b4b933497df9b3
I20260929 00:55:04.741552 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 5568e883283d4f6797b4b933497df9b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260929 00:55:04.741768 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet 5568e883283d4f6797b4b933497df9b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260929 00:55:04.741746 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet 5568e883283d4f6797b4b933497df9b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41750
I20260929 00:55:04.742147 8499 tablet_replica.cc:330] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: stopping tablet replica
I20260929 00:55:04.742377 8499 raft_consensus.cc:2229] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.742568 8499 raft_consensus.cc:2258] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.742880 8497 tablet_replica.cc:330] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: stopping tablet replica
I20260929 00:55:04.743184 8497 raft_consensus.cc:2229] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:04.743894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8036
I20260929 00:55:04.744477 8497 raft_consensus.cc:2258] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.744808 8498 tablet_replica.cc:330] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: stopping tablet replica
I20260929 00:55:04.745031 8498 raft_consensus.cc:2229] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:04.746546 8497 ts_tablet_manager.cc:1905] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.746820 8498 raft_consensus.cc:2258] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:04.749162 8498 ts_tablet_manager.cc:1905] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:04.751991 8499 ts_tablet_manager.cc:1905] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 00:55:04.756425 8497 ts_tablet_manager.cc:1918] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.756623 8497 log.cc:1196] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/5568e883283d4f6797b4b933497df9b3
I20260929 00:55:04.757082 8497 ts_tablet_manager.cc:1939] T 5568e883283d4f6797b4b933497df9b3 P 4a5188ac31e94d909ce8a0190a2b2c69: Deleting consensus metadata
I20260929 00:55:04.758265 7951 catalog_manager.cc:4841] TS 4a5188ac31e94d909ce8a0190a2b2c69 (127.0.97.2:40503): tablet 5568e883283d4f6797b4b933497df9b3 (table test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]) successfully deleted
I20260929 00:55:04.760738 8499 ts_tablet_manager.cc:1918] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.760957 8499 log.cc:1196] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/5568e883283d4f6797b4b933497df9b3
I20260929 00:55:04.761420 8499 ts_tablet_manager.cc:1939] T 5568e883283d4f6797b4b933497df9b3 P 0ab4c4752cdb44429c8d6242ef531954: Deleting consensus metadata
I20260929 00:55:04.762269 8498 ts_tablet_manager.cc:1918] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 00:55:04.762451 8498 log.cc:1196] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/5568e883283d4f6797b4b933497df9b3
I20260929 00:55:04.762630 7950 catalog_manager.cc:4841] TS 0ab4c4752cdb44429c8d6242ef531954 (127.0.97.3:40765): tablet 5568e883283d4f6797b4b933497df9b3 (table test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]) successfully deleted
I20260929 00:55:04.762892 8498 ts_tablet_manager.cc:1939] T 5568e883283d4f6797b4b933497df9b3 P 746145d08c714ac39a5008efe23c232f: Deleting consensus metadata
I20260929 00:55:04.763983 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.745222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.0.97.1:35809)
0929 00:55:04.745528 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.745534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.745550 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:55:04.745604 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.745607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.745660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.745774 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:04.747290 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.748431 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.750178 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.750899 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.751039 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.756004 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:04.756024 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:04.756046 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:04.756093 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:04.760353 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.761277 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.761289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.761293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.761456 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.761940 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.761945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.761948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.762384 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:04.762752 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.763307 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.763553 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 00:55:04.763928 7949 catalog_manager.cc:4841] TS 746145d08c714ac39a5008efe23c232f (127.0.97.1:35809): tablet 5568e883283d4f6797b4b933497df9b3 (table test-table [id=f0fad3b45b6f4825b7df4e2165b5494d]) successfully deleted
I20260929 00:55:04.764747 8167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:04.910167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8036
I20260929 00:55:04.934127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8176
Sep 29 00:55:04 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 00:55:04.948050 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:04.934828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.97.2:40503)
0929 00:55:04.935007 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:04.935055 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:04.935081 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:04.935295 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:04.935300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:04.935359 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:04.935473 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:04.936729 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.937452 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.938616 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:04.938999 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:04.939131 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:04.943039 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:04.943051 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:04.943065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:04.943091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:04.945857 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.946407 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.946413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.946415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.946526 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:04.946808 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:04.946811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:04.946813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:04.947139 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:04.947358 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:04.947542 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:55:04.947691 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:55:04.948832 8306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:05.071496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8176
I20260929 00:55:05.093636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8315
Sep 29 00:55:05 dist-test-slave-pzsx krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 00:55:05.107967 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:05.094339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.0.97.3:40765)
0929 00:55:05.094845 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:05.094849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:05.094864 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:05.094914 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:05.094917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:05.094973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:05.095090 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:05.096294 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:05.097162 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:05.098320 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:05.098772 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:05.098899 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:05.102926 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:05.102940 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:05.102957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:05.102987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:05.105757 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:05.106419 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:05.106426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:05.106428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:05.106551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:05.106831 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:05.106834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:05.106836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:05.107137 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:05.107352 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:05.107552 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 00:55:05.107702 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
I20260929 00:55:05.108700 8445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:05.203281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8315
I20260929 00:55:05.220476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 7933
I20260929 00:55:05.221580 8005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:05.334192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 7933
2026-09-29T00:55:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5533 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:05 dist-test-slave-pzsx krb5kdc[8572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:05 dist-test-slave-pzsx krb5kdc[8572](info): set up 2 sockets
Sep 29 00:55:05 dist-test-slave-pzsx krb5kdc[8572](info): commencing operation
krb5kdc: starting...
W20260929 00:55:06.421516 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:55:06 dist-test-slave-pzsx krb5kdc[8572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:55:06.630364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46091
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:06.946372 8585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:06.946650 8585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:06.946826 8585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:06.953970 8585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:06.954078 8585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:06.954200 8585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:06.954281 8585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:06.954345 8585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:06.963855 8585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:06.965166 8585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:06.966442 8585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:06.978039 8593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:06.978152 8590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:06.978184 8591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:06.979238 8585 server_base.cc:1034] running on GCE node
I20260929 00:55:06.979995 8585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:06.981501 8585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:06.982721 8585 hybrid_clock.cc:648] HybridClock initialized: now 1790643306982682 us; error 33 us; skew 500 ppm
I20260929 00:55:06.983088 8585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T00:55:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1712 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:07 dist-test-slave-pzsx krb5kdc[8599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:07 dist-test-slave-pzsx krb5kdc[8599](info): set up 2 sockets
Sep 29 00:55:07 dist-test-slave-pzsx krb5kdc[8599](info): commencing operation
krb5kdc: starting...
W20260929 00:55:08.153435 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:55:08 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:55:08.370275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46577
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:08.693209 8612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:08.693499 8612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:08.693665 8612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:08.700701 8612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:08.700840 8612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:08.700948 8612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:08.701040 8612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:08.701117 8612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:08.710889 8612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:08.712224 8612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:08.713515 8612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:08.725194 8620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:08.725101 8617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:08.725131 8618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:08.725939 8612 server_base.cc:1034] running on GCE node
I20260929 00:55:08.726627 8612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:08.727941 8612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:08.729133 8612 hybrid_clock.cc:648] HybridClock initialized: now 1790643308729089 us; error 27 us; skew 500 ppm
I20260929 00:55:08.729470 8612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:08 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:08.736188 8612 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:55:08.737715 8612 webserver.cc:460] Webserver started at http://127.0.97.62:42013/ using document root <none> and password file <none>
I20260929 00:55:08.738404 8612 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:08.738529 8612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:08.738848 8612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:08.741818 8612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "779a8c89b741495896a344ff11040516"
format_stamp: "Formatted at 2026-09-29 00:55:08 on dist-test-slave-pzsx"
server_key: "c57aa3f1ccb34342c88f47cf187bfe03"
server_key_iv: "2647e6de97d2add68151ee51f266715d"
server_key_version: "encryptionkey@0"
I20260929 00:55:08.742604 8612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "779a8c89b741495896a344ff11040516"
format_stamp: "Formatted at 2026-09-29 00:55:08 on dist-test-slave-pzsx"
server_key: "c57aa3f1ccb34342c88f47cf187bfe03"
server_key_iv: "2647e6de97d2add68151ee51f266715d"
server_key_version: "encryptionkey@0"
I20260929 00:55:08.747475 8612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 00:55:08.750733 8627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:08.751976 8612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:55:08.752151 8612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "779a8c89b741495896a344ff11040516"
format_stamp: "Formatted at 2026-09-29 00:55:08 on dist-test-slave-pzsx"
server_key: "c57aa3f1ccb34342c88f47cf187bfe03"
server_key_iv: "2647e6de97d2add68151ee51f266715d"
server_key_version: "encryptionkey@0"
I20260929 00:55:08.752336 8612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:08.766685 8612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:08.770391 8612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:08.770650 8612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:08.788522 8612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46577
I20260929 00:55:08.788601 8678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46577 every 8 connection(s)
I20260929 00:55:08.789950 8612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:08.793421 8679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:08.800396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8612
I20260929 00:55:08.800618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:55:08.801405 388 external_mini_cluster.cc:1428] Setting key ef5089dbe6996968e2a56de53251d429
I20260929 00:55:08.804015 8679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Bootstrap starting.
I20260929 00:55:08.807425 8679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:08.808717 8679 log.cc:824] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:08.811694 8679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: No bootstrap required, opened a new log
Sep 29 00:55:08 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:08.820233 8679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:08.820595 8679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:08.820691 8679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779a8c89b741495896a344ff11040516, State: Initialized, Role: FOLLOWER
I20260929 00:55:08.821334 8679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:08.821611 8679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:08.821759 8679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:08.821944 8679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:08.824751 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:08.803320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.0.97.62:46577)
0929 00:55:08.804056 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:08.804067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:08.804105 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:08.806010 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:08.806032 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:08.806396 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:08.806696 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:08.808551 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:08.810274 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:08.811964 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:08.812650 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:08.812789 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:08.817153 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:08.817161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:08.817175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:08.817211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:08.820189 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:08.821048 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:08.821056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:08.821064 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:08.821256 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:08.821621 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:08.821629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:08.821633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:08.822353 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:08.822667 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:08.822985 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 00:55:08.823398 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":337,"threads_started":1}
I20260929 00:55:08.825208 8679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:08.825766 8679 leader_election.cc:302] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779a8c89b741495896a344ff11040516; no voters:
I20260929 00:55:08.826426 8679 leader_election.cc:288] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:08.826540 8684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:08.827993 8684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 LEADER]: Becoming Leader. State: Replica: 779a8c89b741495896a344ff11040516, State: Running, Role: LEADER
I20260929 00:55:08.828487 8684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:08.828752 8679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:08.831450 8686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779a8c89b741495896a344ff11040516. Latest consensus state: current_term: 1 leader_uuid: "779a8c89b741495896a344ff11040516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } } }
I20260929 00:55:08.831467 8685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "779a8c89b741495896a344ff11040516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } } }
I20260929 00:55:08.831897 8686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:08.831926 8685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:08.835325 8690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:08.840337 8690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:08.847836 8690 catalog_manager.cc:1347] Generated new cluster ID: 61ab8be6236c431180523a252b8c96b7
I20260929 00:55:08.847942 8690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:08.857303 8690 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:08.858554 8690 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:08.865451 8690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Generated new TSK 0
I20260929 00:55:08.866282 8690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:55:08.932340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:09.299652 8705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:09.299922 8705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:09.300024 8705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:09.307461 8705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:09.307567 8705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:09.307685 8705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:09.317378 8705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:09.318667 8705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:09.319872 8705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:09.333285 8710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:09.333247 8711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:09.333321 8713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:09.334090 8705 server_base.cc:1034] running on GCE node
I20260929 00:55:09.334790 8705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:09.335733 8705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:09.336912 8705 hybrid_clock.cc:648] HybridClock initialized: now 1790643309336849 us; error 55 us; skew 500 ppm
I20260929 00:55:09.337203 8705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:09 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:09.345363 8705 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:55:09.346748 8705 webserver.cc:460] Webserver started at http://127.0.97.1:44527/ using document root <none> and password file <none>
I20260929 00:55:09.347415 8705 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:09.347551 8705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:09.347853 8705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:09.350800 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "145116d492db4029a7f6206126f01ddd"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "a1c9d73b272e4a2449c0267c92299d73"
server_key_iv: "ca792fdbdc6de68b08b9e29b6ec04903"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.351529 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "145116d492db4029a7f6206126f01ddd"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "a1c9d73b272e4a2449c0267c92299d73"
server_key_iv: "ca792fdbdc6de68b08b9e29b6ec04903"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.356439 8705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 00:55:09.359685 8720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.360955 8705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 00:55:09.361110 8705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "145116d492db4029a7f6206126f01ddd"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "a1c9d73b272e4a2449c0267c92299d73"
server_key_iv: "ca792fdbdc6de68b08b9e29b6ec04903"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.361300 8705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:09.373930 8705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:09.378168 8705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:09.378451 8705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:09.379338 8705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:09.380694 8705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:09.380764 8705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.380909 8705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:09.380982 8705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.412721 8705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41909
I20260929 00:55:09.412762 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41909 every 8 connection(s)
I20260929 00:55:09.413978 8705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:09.416301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8705
I20260929 00:55:09.416523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:09.417307 388 external_mini_cluster.cc:1428] Setting key 8be3fd110d04600e63ea0c56b803b759
Sep 29 00:55:09 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:09.435456 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.416660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55999 (local address 127.0.97.62:46577)
0929 00:55:09.417110 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:09.417114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:09.418329 (+ 1215us) server_negotiation.cc:407] Connection header received
0929 00:55:09.420549 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:09.420553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:09.420649 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:09.420762 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:09.422278 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.422994 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.424549 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.424947 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.425091 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:09.428547 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:09.428557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:09.428565 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:09.428609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:09.432190 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:09.432944 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:09.432953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:09.432959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:09.433111 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:09.433552 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:09.433560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:09.433566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:09.433952 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:09.434113 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:09.434842 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 00:55:09.435058 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260929 00:55:09.436610 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.417278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:55999)
0929 00:55:09.418163 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:09.418219 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:09.420253 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:09.420998 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:09.421007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:09.421538 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:09.422107 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:09.422124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.423140 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.423145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:09.424380 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:09.424391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.425179 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.425187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:09.425433 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:09.426138 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:09.426159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:09.428269 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:09.432333 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:09.432340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:09.432353 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:09.432832 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:09.433231 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:09.433234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:09.433244 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:09.433418 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:09.434098 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:09.434104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:09.434425 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 00:55:09.434738 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 00:55:09.435328 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":329,"threads_started":1}
I20260929 00:55:09.438134 8833 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46577
I20260929 00:55:09.438465 8833 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:55:09.439193 8833 heartbeater.cc:511] Master 127.0.97.62:46577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:55:09.441349 8644 ts_manager.cc:194] Registered new tserver with Master: 145116d492db4029a7f6206126f01ddd (127.0.97.1:41909)
I20260929 00:55:09.442751 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:55:09.465456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:09.781240 8839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:09.781514 8839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:09.781606 8839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:09.788846 8839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:09.788949 8839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:09.789139 8839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:09.798632 8839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:09.799795 8839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:09.801142 8839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:09.813360 8847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:09.813412 8844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:09.813388 8845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:09.814370 8839 server_base.cc:1034] running on GCE node
I20260929 00:55:09.814884 8839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:09.815690 8839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:09.816874 8839 hybrid_clock.cc:648] HybridClock initialized: now 1790643309816822 us; error 51 us; skew 500 ppm
I20260929 00:55:09.817165 8839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:09 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:09.820344 8839 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:55:09.821718 8839 webserver.cc:460] Webserver started at http://127.0.97.2:46751/ using document root <none> and password file <none>
I20260929 00:55:09.822379 8839 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:09.822516 8839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:09.822816 8839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:09.825616 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3f288cb9da04657a07fa99600847c3f"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "89ee7ce9bd163c62927c7c4118872b31"
server_key_iv: "2388043b767d254b599f943c0633745e"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.826426 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3f288cb9da04657a07fa99600847c3f"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "89ee7ce9bd163c62927c7c4118872b31"
server_key_iv: "2388043b767d254b599f943c0633745e"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.831307 8839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 00:55:09.834527 8854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.835800 8839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 00:55:09.835958 8839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "a3f288cb9da04657a07fa99600847c3f"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "89ee7ce9bd163c62927c7c4118872b31"
server_key_iv: "2388043b767d254b599f943c0633745e"
server_key_version: "encryptionkey@0"
I20260929 00:55:09.836141 8839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:09.852166 8839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:09.856389 8839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:09.856686 8839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:09.857916 8839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:09.859292 8839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:09.859395 8839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.859514 8839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:09.859593 8839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:09.891274 8839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46569
I20260929 00:55:09.891330 8966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46569 every 8 connection(s)
I20260929 00:55:09.892601 8839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:09.894095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8839
I20260929 00:55:09.894275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:09.894733 388 external_mini_cluster.cc:1428] Setting key a3c456c3973c1648b856566b32ad011b
Sep 29 00:55:09 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:09.916210 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.895059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36281 (local address 127.0.97.62:46577)
0929 00:55:09.895253 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:09.895259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:09.896984 (+ 1725us) server_negotiation.cc:407] Connection header received
0929 00:55:09.899997 (+ 3013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:09.900001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:09.900065 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:09.900166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:09.902345 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.903023 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.905206 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.905941 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.906084 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:09.909464 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:09.909473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:09.909481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:09.909525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:09.913070 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:09.913775 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:09.913784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:09.913791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:09.913954 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:09.914357 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:09.914365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:09.914371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:09.914794 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:09.914947 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:09.915640 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 00:55:09.915861 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:55:09.917271 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:09.895586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:36281)
0929 00:55:09.896823 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:09.896878 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:09.899604 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:09.900445 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:09.900459 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:09.901280 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:09.902088 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:09.902114 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.903187 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.903196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:09.905058 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:09.905069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:09.906154 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:09.906159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:09.906355 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:09.907035 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:09.907057 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:09.909220 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:09.913195 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:09.913202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:09.913216 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:09.913610 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:09.914056 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:09.914059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:09.914062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:09.914246 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:09.914939 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:09.914945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:09.915282 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 00:55:09.915542 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 00:55:09.916088 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":715,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:55:09.918821 8967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46577
I20260929 00:55:09.919154 8967 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:09.919911 8967 heartbeater.cc:511] Master 127.0.97.62:46577 requested a full tablet report, sending...
I20260929 00:55:09.921257 8644 ts_manager.cc:194] Registered new tserver with Master: a3f288cb9da04657a07fa99600847c3f (127.0.97.2:46569)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:55:09.922197 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:55:09.945606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:10.266894 8972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:10.267189 8972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:10.267297 8972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:10.274863 8972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:10.274969 8972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:10.275128 8972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:10.284793 8972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:10.285957 8972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:10.287163 8972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:10.299618 8980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:10.299703 8977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:10.299731 8978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:10.300542 8972 server_base.cc:1034] running on GCE node
I20260929 00:55:10.301182 8972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:10.302042 8972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:10.303254 8972 hybrid_clock.cc:648] HybridClock initialized: now 1790643310303209 us; error 39 us; skew 500 ppm
I20260929 00:55:10.303565 8972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:10 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:10.306563 8972 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:55:10.307898 8972 webserver.cc:460] Webserver started at http://127.0.97.3:38217/ using document root <none> and password file <none>
I20260929 00:55:10.308624 8972 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:10.308794 8972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:10.309159 8972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:10.311877 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "701d93d79c90495b8a8022ecb0a9c4c9"
format_stamp: "Formatted at 2026-09-29 00:55:10 on dist-test-slave-pzsx"
server_key: "f13c3be7d0e6c6b7f2212dfb077bf710"
server_key_iv: "b2cce81be49b239322fb8932a635861d"
server_key_version: "encryptionkey@0"
I20260929 00:55:10.312626 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "701d93d79c90495b8a8022ecb0a9c4c9"
format_stamp: "Formatted at 2026-09-29 00:55:10 on dist-test-slave-pzsx"
server_key: "f13c3be7d0e6c6b7f2212dfb077bf710"
server_key_iv: "b2cce81be49b239322fb8932a635861d"
server_key_version: "encryptionkey@0"
I20260929 00:55:10.317338 8972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:55:10.320562 8987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:10.321805 8972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:55:10.321933 8972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "701d93d79c90495b8a8022ecb0a9c4c9"
format_stamp: "Formatted at 2026-09-29 00:55:10 on dist-test-slave-pzsx"
server_key: "f13c3be7d0e6c6b7f2212dfb077bf710"
server_key_iv: "b2cce81be49b239322fb8932a635861d"
server_key_version: "encryptionkey@0"
I20260929 00:55:10.322099 8972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:10.352133 8972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:10.356448 8972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:10.356760 8972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:10.357741 8972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:10.358949 8972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:10.359018 8972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:10.359105 8972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:10.359161 8972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:10.390760 8972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34297
I20260929 00:55:10.390828 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34297 every 8 connection(s)
I20260929 00:55:10.392002 8972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:10.395109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 8972
I20260929 00:55:10.395262 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:10.395845 388 external_mini_cluster.cc:1428] Setting key db1611cdfaccec9dd80b07d12d51dd3a
Sep 29 00:55:10 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:10.411475 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.394513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40669 (local address 127.0.97.62:46577)
0929 00:55:10.394719 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:10.394725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:10.396073 (+ 1348us) server_negotiation.cc:407] Connection header received
0929 00:55:10.398457 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:10.398463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:10.398542 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:10.398644 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:10.400109 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.400881 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.402303 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.402673 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.402756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:10.405885 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:10.405890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:10.405893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:10.405921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:10.408406 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.409254 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.409259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.409268 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.409387 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.409825 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.409830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.409833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.410082 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:10.410230 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:10.410899 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 00:55:10.411115 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:55:10.412428 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.395028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:40669)
0929 00:55:10.395895 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:10.395945 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:10.398182 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:10.398903 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:10.398911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:10.399407 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:10.399977 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:10.399993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.400989 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.400992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:10.402194 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:10.402204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.402848 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.402852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:10.403041 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:10.403618 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:10.403638 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:10.405710 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:10.408554 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:10.408566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:10.408578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:10.409144 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:10.409517 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:10.409522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:10.409526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:10.409704 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:10.410204 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:10.410212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:10.410524 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 00:55:10.410790 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 00:55:10.411311 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":365,"threads_started":1}
I20260929 00:55:10.413766 9100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46577
I20260929 00:55:10.414041 9100 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:10.414705 9100 heartbeater.cc:511] Master 127.0.97.62:46577 requested a full tablet report, sending...
I20260929 00:55:10.416086 8644 ts_manager.cc:194] Registered new tserver with Master: 701d93d79c90495b8a8022ecb0a9c4c9 (127.0.97.3:34297)
I20260929 00:55:10.417178 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40669
I20260929 00:55:10.422560 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260929 00:55:10.445505 8833 heartbeater.cc:503] Master 127.0.97.62:46577 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 00:55:10.553532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8705
Sep 29 00:55:10 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:55:10.569756 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.554630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.97.1:41909)
0929 00:55:10.555257 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:10.555264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:10.555318 (+ 54us) server_negotiation.cc:407] Connection header received
0929 00:55:10.555378 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:10.555385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:10.555686 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:10.555820 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:10.557292 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.558288 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.559444 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.560007 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.560102 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:10.563835 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:10.563846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:10.563860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:10.563895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:10.567134 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.567799 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.567806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.567809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.567950 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.568275 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.568278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.568280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.568682 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:10.568920 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:10.569172 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 00:55:10.569363 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":318,"threads_started":1}
I20260929 00:55:10.571053 8828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:10.653689 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8705
I20260929 00:55:10.669867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8839
Sep 29 00:55:10 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:55:10.684871 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.670759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.0.97.2:46569)
0929 00:55:10.671273 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:10.671280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:10.671304 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:55:10.671363 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:10.671370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:10.671577 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:10.671748 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:10.672994 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.673954 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.675210 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.675757 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.675891 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:10.679890 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:10.679900 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:10.679914 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:10.679945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:10.682614 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.683174 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.683181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.683183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.683290 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.683572 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.683574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.683576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.683874 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:10.684094 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:10.684278 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:55:10.684451 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260929 00:55:10.686138 8962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:10.768354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8839
I20260929 00:55:10.784035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8972
Sep 29 00:55:10 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:55:10.800348 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:10.784988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.97.3:34297)
0929 00:55:10.785607 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:10.785614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:10.785643 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:55:10.785702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:10.785708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:10.785917 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:10.786101 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:10.787408 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.788379 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.789773 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:10.790323 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:10.790454 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:10.795068 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:10.795079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:10.795093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:10.795124 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:10.797679 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.798318 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.798325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.798327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.798448 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:10.798798 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:10.798805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:10.798808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:10.799157 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:10.799473 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:10.799759 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 00:55:10.800012 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
I20260929 00:55:10.801635 9095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:10.883419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8972
I20260929 00:55:10.899325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 8612
I20260929 00:55:10.900445 8674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:11.001814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 8612
I20260929 00:55:11.022131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46577
--webserver_interface=127.0.97.62
--webserver_port=42013
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:11.338425 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:11.338697 9117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:11.338802 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:11.345865 9117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:11.345976 9117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:11.346021 9117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:11.346081 9117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:11.346124 9117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:11.355585 9117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:11.356805 9117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:11.358031 9117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:11.369553 9123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:11.369526 9122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:11.369493 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:11.370478 9117 server_base.cc:1034] running on GCE node
I20260929 00:55:11.371208 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:11.372562 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:11.373778 9117 hybrid_clock.cc:648] HybridClock initialized: now 1790643311373720 us; error 49 us; skew 500 ppm
I20260929 00:55:11.374120 9117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:11.377120 9117 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:55:11.378492 9117 webserver.cc:460] Webserver started at http://127.0.97.62:42013/ using document root <none> and password file <none>
I20260929 00:55:11.379159 9117 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:11.379307 9117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:11.384648 9117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:11.387257 9132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:11.388468 9117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:55:11.388690 9117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "779a8c89b741495896a344ff11040516"
format_stamp: "Formatted at 2026-09-29 00:55:08 on dist-test-slave-pzsx"
server_key: "c57aa3f1ccb34342c88f47cf187bfe03"
server_key_iv: "2647e6de97d2add68151ee51f266715d"
server_key_version: "encryptionkey@0"
I20260929 00:55:11.389631 9117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:11.405288 9117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:11.408996 9117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:11.409287 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:11.427621 9117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46577
I20260929 00:55:11.427644 9183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46577 every 8 connection(s)
I20260929 00:55:11.429239 9117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:11.430953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9117
I20260929 00:55:11.431707 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41909
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46577
--webserver_port=44527
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:55:11.436239 9184 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:55:11.440590 9184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Bootstrap starting.
I20260929 00:55:11.469026 9184 log.cc:824] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:11.479842 9184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:55:11.480489 9184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Bootstrap complete.
I20260929 00:55:11.494134 9184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:11.495327 9184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779a8c89b741495896a344ff11040516, State: Initialized, Role: FOLLOWER
I20260929 00:55:11.496029 9184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:11.496307 9184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:11.496457 9184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:11.496690 9184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:55:11.503203 9184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:11.503808 9184 leader_election.cc:302] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779a8c89b741495896a344ff11040516; no voters:
I20260929 00:55:11.504426 9184 leader_election.cc:288] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:55:11.504590 9188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:55:11.505740 9188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [term 2 LEADER]: Becoming Leader. State: Replica: 779a8c89b741495896a344ff11040516, State: Running, Role: LEADER
I20260929 00:55:11.506282 9188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } }
I20260929 00:55:11.506486 9184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:11.508911 9189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "779a8c89b741495896a344ff11040516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } } }
I20260929 00:55:11.508989 9190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779a8c89b741495896a344ff11040516. Latest consensus state: current_term: 2 leader_uuid: "779a8c89b741495896a344ff11040516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779a8c89b741495896a344ff11040516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46577 } } }
I20260929 00:55:11.509286 9190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:11.509290 9189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:11.520280 9195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:11.525960 9195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:11.527328 9195 catalog_manager.cc:1259] Loaded cluster ID: 61ab8be6236c431180523a252b8c96b7
I20260929 00:55:11.527482 9195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:11.531548 9195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:11.532987 9195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 779a8c89b741495896a344ff11040516: Loaded TSK: 0
I20260929 00:55:11.534518 9195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:55:11.792660 9186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:11.792939 9186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:11.793083 9186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:11.800432 9186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:11.800612 9186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:11.800817 9186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:11.810706 9186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44527
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:11.812215 9186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:11.813541 9186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:11.827939 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:11.827945 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:11.828629 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:11.828863 9186 server_base.cc:1034] running on GCE node
I20260929 00:55:11.829495 9186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:11.830399 9186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:11.831679 9186 hybrid_clock.cc:648] HybridClock initialized: now 1790643311831625 us; error 41 us; skew 500 ppm
I20260929 00:55:11.832031 9186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643311, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:11.835130 9186 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 00:55:11.836650 9186 webserver.cc:460] Webserver started at http://127.0.97.1:44527/ using document root <none> and password file <none>
I20260929 00:55:11.837318 9186 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:11.837456 9186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:11.842705 9186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 00:55:11.845510 9220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:11.846854 9186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:11.847030 9186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "145116d492db4029a7f6206126f01ddd"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "a1c9d73b272e4a2449c0267c92299d73"
server_key_iv: "ca792fdbdc6de68b08b9e29b6ec04903"
server_key_version: "encryptionkey@0"
I20260929 00:55:11.847935 9186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:11.866194 9186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:11.870554 9186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:11.870827 9186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:11.871786 9186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:11.873145 9186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:11.873260 9186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:11.873379 9186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:11.873466 9186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:11.905390 9186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41909
I20260929 00:55:11.905439 9332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41909 every 8 connection(s)
I20260929 00:55:11.906821 9186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:11.916926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9186
I20260929 00:55:11.917508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46569
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46577
--webserver_port=46751
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:11.924939 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.910068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:43057)
0929 00:55:11.910822 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.910863 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:11.912632 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:11.913891 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:11.913898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:11.914389 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:11.914900 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.914916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.916414 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.916417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.917741 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.917754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.919211 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.919216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.919396 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.920039 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:11.920059 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:11.923679 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":347,"threads_started":1}
W20260929 00:55:11.924968 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:11.909947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43057 (local address 127.0.97.62:46577)
0929 00:55:11.910657 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:11.910675 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:11.910988 (+ 313us) server_negotiation.cc:407] Connection header received
0929 00:55:11.913127 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:11.913140 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:11.913464 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:11.913739 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:11.915056 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.916273 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.917877 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.919019 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.919207 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.924098 (+ 4891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43057: BlockingRecv error: recv got EOF from 127.0.97.1:43057 (error 108)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":233,"threads_started":1}
W20260929 00:55:11.925805 9333 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46577: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:11.934309 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.926380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:35563)
0929 00:55:11.926552 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.926570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:11.926669 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:11.927030 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:11.927033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:11.927310 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:11.927434 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.927442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.928908 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.928912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.929781 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.929791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.930792 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.930796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.930911 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.931456 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:11.931474 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:11.934102 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 00:55:11.934661 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:11.926554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35563 (local address 127.0.97.62:46577)
0929 00:55:11.926739 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:11.926747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:11.926767 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:11.926817 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:11.926825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:11.926902 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:11.926996 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:11.927564 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.928792 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.929911 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.930659 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.930786 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.934378 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35563: BlockingRecv error: recv got EOF from 127.0.97.1:35563 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:11 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:11.943109 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:11.935142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:54567)
0929 00:55:11.935331 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:11.935362 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:11.935452 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:11.935818 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:11.935821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:11.936063 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:11.936190 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.936197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.937681 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.937685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.938570 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:11.938580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.939660 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.939664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:11.939776 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.940335 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:11.940351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:11.942910 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 00:55:11.943387 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:11.935291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54567 (local address 127.0.97.62:46577)
0929 00:55:11.935511 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:11.935519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:11.935539 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:11.935592 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:11.935599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:11.935693 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:11.935805 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:11.936322 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.937553 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.938701 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:11.939520 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:11.939671 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:11.943130 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54567: BlockingRecv error: recv got EOF from 127.0.97.1:54567 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 00:55:11.943470 9333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:55:12.245983 9337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:12.246297 9337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:12.246388 9337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:12.253614 9337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:12.253723 9337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:12.253840 9337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:12.263278 9337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46751
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:12.264539 9337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:12.265774 9337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:12.278270 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:12.278321 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:12.278344 9345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:12.279862 9337 server_base.cc:1034] running on GCE node
I20260929 00:55:12.280449 9337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:12.281421 9337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:12.282641 9337 hybrid_clock.cc:648] HybridClock initialized: now 1790643312282598 us; error 54 us; skew 500 ppm
I20260929 00:55:12.282945 9337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643312, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:12.286446 9337 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 00:55:12.287757 9337 webserver.cc:460] Webserver started at http://127.0.97.2:46751/ using document root <none> and password file <none>
I20260929 00:55:12.288403 9337 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:12.288563 9337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:12.293622 9337 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:55:12.296075 9352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.297302 9337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:12.297459 9337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "a3f288cb9da04657a07fa99600847c3f"
format_stamp: "Formatted at 2026-09-29 00:55:09 on dist-test-slave-pzsx"
server_key: "89ee7ce9bd163c62927c7c4118872b31"
server_key_iv: "2388043b767d254b599f943c0633745e"
server_key_version: "encryptionkey@0"
I20260929 00:55:12.298332 9337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:12.312700 9337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:12.316895 9337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:12.317169 9337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:12.318063 9337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:12.319248 9337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:12.319350 9337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.319474 9337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:12.319558 9337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.350639 9337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46569
I20260929 00:55:12.350719 9464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46569 every 8 connection(s)
I20260929 00:55:12.352103 9337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:12.360963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9337
I20260929 00:55:12.361528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34297
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46577
--webserver_port=38217
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:55:12.368587 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.354652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50263 (local address 127.0.97.62:46577)
0929 00:55:12.354819 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.354823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.355986 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 00:55:12.357819 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.357824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.357889 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.357968 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.359343 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.360020 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.361948 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.362700 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.362849 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.368275 (+ 5426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50263: BlockingRecv error: recv got EOF from 127.0.97.2:50263 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:55:12.369100 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.355159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:50263)
0929 00:55:12.355831 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.355876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.357559 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.358168 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.358176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.358674 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.359200 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.359216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.360188 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.360193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.361790 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.361805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.362906 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.362911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.363098 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.363757 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.363784 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.368007 (+ 4223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
W20260929 00:55:12.369796 9465 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46577: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:55:12.379571 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.370517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34557 (local address 127.0.97.62:46577)
0929 00:55:12.370674 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.370678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.370693 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:55:12.370739 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.370743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.370798 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.370875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.371734 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.372406 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.374384 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.374758 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.374856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.379400 (+ 4544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34557: BlockingRecv error: recv got EOF from 127.0.97.2:34557 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:55:12.379611 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.370322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:34557)
0929 00:55:12.370526 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.370547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.370655 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.370926 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.370933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.371389 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.371593 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.371607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.372574 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.372582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.374256 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.374272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.374907 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.374915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.375131 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.375743 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.375769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.379257 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:12.388403 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.380296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:50667)
0929 00:55:12.380565 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.380580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.380660 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.380965 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.380968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.381208 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.381337 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.381344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.382825 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.382828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.383698 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.383719 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.384781 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.384785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.384895 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.385450 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.385466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.388115 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260929 00:55:12.388415 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.380438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50667 (local address 127.0.97.62:46577)
0929 00:55:12.380657 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.380666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.380687 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:55:12.380746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.380752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.380836 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.380954 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.381474 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.382705 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.383836 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.384643 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.384780 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.388271 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50667: BlockingRecv error: recv got EOF from 127.0.97.2:50667 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 00:55:12.388696 9465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:55:12.692103 9468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:12.692404 9468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:12.692572 9468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:12.699717 9468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:12.699818 9468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:12.699927 9468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:12.709339 9468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38217
--tserver_master_addrs=127.0.97.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:12.710525 9468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:12.711740 9468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:12.724725 9474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:12.724833 9476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:12.725052 9473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:12.725580 9468 server_base.cc:1034] running on GCE node
I20260929 00:55:12.726145 9468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:12.726986 9468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:12.728197 9468 hybrid_clock.cc:648] HybridClock initialized: now 1790643312728148 us; error 44 us; skew 500 ppm
I20260929 00:55:12.728488 9468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643312, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:12.731487 9468 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 00:55:12.732803 9468 webserver.cc:460] Webserver started at http://127.0.97.3:38217/ using document root <none> and password file <none>
I20260929 00:55:12.733408 9468 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:12.733510 9468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:12.738426 9468 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 00:55:12.740881 9483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.742098 9468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:12.742225 9468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "701d93d79c90495b8a8022ecb0a9c4c9"
format_stamp: "Formatted at 2026-09-29 00:55:10 on dist-test-slave-pzsx"
server_key: "f13c3be7d0e6c6b7f2212dfb077bf710"
server_key_iv: "b2cce81be49b239322fb8932a635861d"
server_key_version: "encryptionkey@0"
I20260929 00:55:12.743085 9468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:12.767040 9468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:12.771118 9468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:12.771332 9468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:12.772181 9468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:12.773452 9468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:12.773548 9468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.773623 9468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:12.773674 9468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:12.805284 9468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34297
I20260929 00:55:12.805380 9595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34297 every 8 connection(s)
I20260929 00:55:12.806739 9468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:12.816000 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9468
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 00:55:12.822785 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.809265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34677 (local address 127.0.97.62:46577)
0929 00:55:12.809424 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.809429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.810592 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 00:55:12.812362 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.812367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.812466 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.812603 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.814068 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.814746 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.816242 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.816852 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.816988 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.822520 (+ 5532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34677: BlockingRecv error: recv got EOF from 127.0.97.3:34677 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:55:12.823365 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.809752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:34677)
0929 00:55:12.810438 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.810479 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.812097 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.812816 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.812824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.813300 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.813903 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.813920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.814890 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.814894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.816081 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.816092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.817088 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.817098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.817416 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.818262 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.818294 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.822316 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":266,"threads_started":1}
W20260929 00:55:12.824057 9596 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46577: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:12.832304 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.824674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:56545)
0929 00:55:12.824845 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.824864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.824973 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.825252 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.825259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.825674 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.825869 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.825883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.826779 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.826787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.827841 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.827852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.828856 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.828860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.828977 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.829455 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.829470 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.832019 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:55:12.832433 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.824717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56545 (local address 127.0.97.62:46577)
0929 00:55:12.824868 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.824871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.824915 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:55:12.825076 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.825079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.825128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.825200 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.825998 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.826650 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.827969 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.828725 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.828854 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.832189 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56545: BlockingRecv error: recv got EOF from 127.0.97.3:56545 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:55:12.833335 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.817049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.97.62:46577)
0929 00:55:12.817779 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.817790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.817816 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:12.817851 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.817873 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.817942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.818065 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.820614 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.821859 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.823378 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.823893 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.824017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.826938 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:12.826945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:12.826959 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:12.826994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:12.830035 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:12.830720 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:12.830726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:12.830732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:12.830904 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:12.831314 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:12.831320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:12.831327 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:12.831778 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:12.832089 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:12.832560 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 00:55:12.832901 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":489,"threads_started":1}
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:12.840306 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.833108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:54041)
0929 00:55:12.833270 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.833291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.833389 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.833701 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.833704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.833943 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.834085 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.834091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.835238 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.835243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.836297 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.836308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.837079 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.837085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.837254 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.837694 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.837708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.840115 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:55:12.840556 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.833193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54041 (local address 127.0.97.62:46577)
0929 00:55:12.833425 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.833433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.833475 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:55:12.833523 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.833528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.833583 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.833673 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.834201 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.835100 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.836419 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.836950 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.837061 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.840283 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54041: BlockingRecv error: recv got EOF from 127.0.97.3:54041 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260929 00:55:12.840654 9596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:12 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:12.951717 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:12.943978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:60689)
0929 00:55:12.944538 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:12.944558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:12.944665 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:12.944932 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:12.944935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:12.945361 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:12.945553 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.945562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.946507 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.946511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.947520 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:12.947531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.948180 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.948184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:12.948310 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.948850 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:12.948866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:12.951452 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
W20260929 00:55:12.951864 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:12.944017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60689 (local address 127.0.97.62:46577)
0929 00:55:12.944212 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:12.944216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:12.944621 (+ 405us) server_negotiation.cc:407] Connection header received
0929 00:55:12.944751 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:12.944755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:12.944813 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:12.944890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:12.945663 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.946385 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.947641 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:12.948039 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:12.948131 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:12.951674 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60689: BlockingRecv error: recv got EOF from 127.0.97.1:60689 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:13.397091 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.389342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:50159)
0929 00:55:13.389863 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.389883 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:13.389994 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:13.390271 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:13.390274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:13.390698 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:13.390897 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.390906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.391803 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.391806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.392881 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.392892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.393595 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.393600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.393759 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.394349 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:13.394366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:13.396881 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":280,"threads_started":1}
W20260929 00:55:13.397253 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:13.389399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50159 (local address 127.0.97.62:46577)
0929 00:55:13.389605 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:13.389613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:13.389954 (+ 341us) server_negotiation.cc:407] Connection header received
0929 00:55:13.390084 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:13.390088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:13.390145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:13.390226 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:13.391011 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.391688 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.393004 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.393412 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.393497 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.397047 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50159: BlockingRecv error: recv got EOF from 127.0.97.2:50159 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:13.849409 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.841228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:50201)
0929 00:55:13.841732 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.841747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:13.841857 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:13.842134 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:13.842137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:13.842561 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:13.842759 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.842767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.843712 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.843715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.844763 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.844775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.845457 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.845461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.845576 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.846167 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:13.846184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:13.849078 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
W20260929 00:55:13.849584 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:13.841361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50201 (local address 127.0.97.62:46577)
0929 00:55:13.841540 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:13.841544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:13.841819 (+ 275us) server_negotiation.cc:407] Connection header received
0929 00:55:13.841948 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:13.841952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:13.842011 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:13.842096 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:13.842872 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.843590 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.844885 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.845294 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.845385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.849298 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50201: BlockingRecv error: recv got EOF from 127.0.97.3:50201 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:13 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:13.960711 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:13.952740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:35667)
0929 00:55:13.953255 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:13.953275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:13.953424 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:13.953779 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:13.953784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:13.954193 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:13.954342 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.954351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.955323 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.955331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.956311 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:13.956342 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.957110 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.957114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:13.957224 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.957811 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:13.957827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:13.960425 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":303,"threads_started":1}
W20260929 00:55:13.960880 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:13.952798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35667 (local address 127.0.97.62:46577)
0929 00:55:13.953009 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:13.953018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:13.953353 (+ 335us) server_negotiation.cc:407] Connection header received
0929 00:55:13.953566 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:13.953570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:13.953648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:13.953746 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:13.954474 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.955206 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.956477 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:13.956962 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:13.957057 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:13.960641 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35667: BlockingRecv error: recv got EOF from 127.0.97.1:35667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:14.409478 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.398133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:55735)
0929 00:55:14.398656 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.398676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:14.398836 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:14.399175 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:14.399180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:14.399585 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:14.399726 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.399734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.400777 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.400783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.401795 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.401806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.402528 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.402534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.402684 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.406551 (+ 3867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:14.406569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:14.409244 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
W20260929 00:55:14.409611 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:14.398204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55735 (local address 127.0.97.62:46577)
0929 00:55:14.398447 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:14.398456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:14.398769 (+ 313us) server_negotiation.cc:407] Connection header received
0929 00:55:14.398972 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:14.398976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:14.399046 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:14.399141 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:14.399867 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.400647 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.401934 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.402369 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.402501 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.409424 (+ 6923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55735: BlockingRecv error: recv got EOF from 127.0.97.2:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:14.858582 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.850338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:52675)
0929 00:55:14.850889 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.850905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:14.851028 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:14.851361 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:14.851364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:14.851732 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:14.851865 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.851872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.853005 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.853010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.854236 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.854246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.854897 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.854903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.855077 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.855619 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:14.855636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:14.858240 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":318,"threads_started":1}
W20260929 00:55:14.858700 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:14.850475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52675 (local address 127.0.97.62:46577)
0929 00:55:14.850639 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:14.850643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:14.850970 (+ 327us) server_negotiation.cc:407] Connection header received
0929 00:55:14.851166 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:14.851170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:14.851230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:14.851320 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:14.851985 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.852847 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.854365 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.854747 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.854877 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.858447 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52675: BlockingRecv error: recv got EOF from 127.0.97.3:52675 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:14 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:14.968950 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:14.961512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:37675)
0929 00:55:14.962071 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:14.962091 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:14.962202 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:14.962474 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:14.962478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:14.962806 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:14.962971 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.962979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.963938 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.963941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.964848 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:14.964859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.965459 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.965463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:14.965581 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.966130 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:14.966146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:14.968700 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":291,"threads_started":1}
W20260929 00:55:14.969120 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:14.961596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37675 (local address 127.0.97.62:46577)
0929 00:55:14.961771 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:14.961775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:14.962167 (+ 392us) server_negotiation.cc:407] Connection header received
0929 00:55:14.962298 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:14.962301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:14.962358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:14.962435 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:14.963092 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.963827 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.964961 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:14.965349 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:14.965434 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:14.968865 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37675: BlockingRecv error: recv got EOF from 127.0.97.1:37675 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:15.418900 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.410429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:47173)
0929 00:55:15.410978 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.410999 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:15.411117 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:15.411549 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:15.411554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:15.412007 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:15.412268 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.412282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.413475 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.413483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.414381 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.414392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.415081 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.415087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.415278 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.415879 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:15.415896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:15.418658 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
W20260929 00:55:15.419047 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:15.410506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47173 (local address 127.0.97.62:46577)
0929 00:55:15.410662 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:15.410666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:15.411070 (+ 404us) server_negotiation.cc:407] Connection header received
0929 00:55:15.411255 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:15.411259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:15.411353 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:15.411463 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:15.412414 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.413353 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.414520 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.414935 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.415056 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.418845 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47173: BlockingRecv error: recv got EOF from 127.0.97.2:47173 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:15.867980 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.859636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:47339)
0929 00:55:15.860163 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.860179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:15.860299 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:15.860640 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:15.860644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:15.861118 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:15.861355 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.861369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.862692 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.862696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.863562 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.863572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.864192 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.864196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.864321 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.864972 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:15.864990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:15.867729 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
W20260929 00:55:15.868105 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:15.859732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47339 (local address 127.0.97.62:46577)
0929 00:55:15.859947 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:15.859951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:15.860254 (+ 303us) server_negotiation.cc:407] Connection header received
0929 00:55:15.860399 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:15.860403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:15.860463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:15.860574 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:15.861544 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.862565 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.863687 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.864078 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.864171 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.867921 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47339: BlockingRecv error: recv got EOF from 127.0.97.3:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:15 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:15.977981 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:15.969885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:50187)
0929 00:55:15.970378 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:15.970397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:15.970555 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:15.970900 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:15.970905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:15.971291 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:15.971438 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.971446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.972440 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.972446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.973453 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:15.973464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.974179 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.974184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:15.974376 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.975020 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:15.975064 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:15.977765 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":275,"threads_started":1}
W20260929 00:55:15.978220 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:15.969963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50187 (local address 127.0.97.62:46577)
0929 00:55:15.970220 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:15.970226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:15.970474 (+ 248us) server_negotiation.cc:407] Connection header received
0929 00:55:15.970678 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:15.970683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:15.970764 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:15.970866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:15.971592 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.972305 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.973587 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:15.974013 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:15.974162 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:15.977947 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50187: BlockingRecv error: recv got EOF from 127.0.97.1:50187 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:16.428092 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.419987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:43447)
0929 00:55:16.420542 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.420567 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:16.420734 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:16.421146 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:16.421153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:16.421649 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:16.421833 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.421845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.422793 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.422796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.423726 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.423736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.424350 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.424354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.424485 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.425135 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:16.425152 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:16.427878 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":300,"threads_started":1}
W20260929 00:55:16.428396 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:16.420073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43447 (local address 127.0.97.62:46577)
0929 00:55:16.420273 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:16.420282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:16.420658 (+ 376us) server_negotiation.cc:407] Connection header received
0929 00:55:16.420889 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:16.420894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:16.420977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:16.421089 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:16.421964 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.422683 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.423850 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.424240 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.424325 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.428150 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43447: BlockingRecv error: recv got EOF from 127.0.97.2:43447 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:16.876711 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.869035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:35297)
0929 00:55:16.869508 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.869524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:16.869629 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:16.869908 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:16.869911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:16.870255 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:16.870402 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.870410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.871351 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.871354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.872295 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.872305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.872911 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.872915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.873034 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.873584 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:16.873601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:16.876417 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":243,"threads_started":1}
W20260929 00:55:16.876876 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:16.869101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35297 (local address 127.0.97.62:46577)
0929 00:55:16.869314 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:16.869320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:16.869602 (+ 282us) server_negotiation.cc:407] Connection header received
0929 00:55:16.869721 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:16.869725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:16.869795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:16.869885 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:16.870511 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.871250 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.872404 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.872799 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.872886 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.876635 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35297: BlockingRecv error: recv got EOF from 127.0.97.3:35297 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:16 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:16.986665 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:16.978836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:51461)
0929 00:55:16.979356 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:16.979375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:16.979479 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:16.979742 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:16.979746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:16.980116 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:16.980276 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.980285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.981219 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.981222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.982147 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:16.982165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.982783 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.982786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:16.982919 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.983467 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:16.983483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:16.986394 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":273,"threads_started":1}
W20260929 00:55:16.986783 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:16.978884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51461 (local address 127.0.97.62:46577)
0929 00:55:16.979065 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:16.979069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:16.979444 (+ 375us) server_negotiation.cc:407] Connection header received
0929 00:55:16.979567 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:16.979571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:16.979628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:16.979709 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:16.980394 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.981108 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.982271 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:16.982674 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:16.982764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:16.986596 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51461: BlockingRecv error: recv got EOF from 127.0.97.1:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:17.438477 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.429058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:34017)
0929 00:55:17.429596 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.429617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:17.429727 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:17.430038 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:17.430041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:17.430435 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:17.430593 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.430602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.432067 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.432071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.433267 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.433283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.434205 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.434209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.434322 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.435170 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:17.435192 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:17.438246 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
W20260929 00:55:17.438678 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:17.429208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34017 (local address 127.0.97.62:46577)
0929 00:55:17.429397 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:17.429403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:17.429695 (+ 292us) server_negotiation.cc:407] Connection header received
0929 00:55:17.429831 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:17.429836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:17.429909 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:17.430008 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:17.430726 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.431939 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.433409 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.434072 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.434194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.438422 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34017: BlockingRecv error: recv got EOF from 127.0.97.2:34017 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:17.885577 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.877741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:37295)
0929 00:55:17.878271 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.878288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:17.878403 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:17.878730 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:17.878733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:17.879084 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:17.879238 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.879246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.880164 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.880167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.881154 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.881164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.881740 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.881744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.881864 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.882421 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:17.882437 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:17.885337 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
W20260929 00:55:17.885674 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:17.877885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37295 (local address 127.0.97.62:46577)
0929 00:55:17.878055 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:17.878060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:17.878362 (+ 302us) server_negotiation.cc:407] Connection header received
0929 00:55:17.878538 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:17.878542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:17.878603 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:17.878687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:17.879354 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.880053 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.881266 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.881633 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.881717 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.885500 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37295: BlockingRecv error: recv got EOF from 127.0.97.3:37295 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:17 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:17.995687 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:17.987666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:46591)
0929 00:55:17.988179 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:17.988198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:17.988352 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:17.988792 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:17.988796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:17.989288 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:17.989477 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.989486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.990556 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.990560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.991438 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:17.991449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.992133 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.992137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:17.992252 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.992899 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:17.992924 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:17.995480 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":293,"threads_started":1}
W20260929 00:55:17.995859 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:17.987740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46591 (local address 127.0.97.62:46577)
0929 00:55:17.987970 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:17.987980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:17.988276 (+ 296us) server_negotiation.cc:407] Connection header received
0929 00:55:17.988482 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:17.988486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:17.988601 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:17.988713 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:17.989583 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.990432 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.991552 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:17.992004 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:17.992091 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:17.995634 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46591: BlockingRecv error: recv got EOF from 127.0.97.1:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:18.447072 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:18.439443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:55567)
0929 00:55:18.439999 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:18.440030 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:18.440141 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:18.440417 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:18.440421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:18.440801 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:18.440943 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:18.440951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.441901 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.441904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:18.442811 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:18.442821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.443435 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.443439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:18.443552 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:18.444125 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:18.444141 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:18.446861 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":329,"threads_started":1}
W20260929 00:55:18.447371 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:18.439483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55567 (local address 127.0.97.62:46577)
0929 00:55:18.439631 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:18.439635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:18.440105 (+ 470us) server_negotiation.cc:407] Connection header received
0929 00:55:18.440233 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:18.440236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:18.440296 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:18.440374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:18.441073 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.441793 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.442922 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.443314 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.443401 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:18.447061 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55567: BlockingRecv error: recv got EOF from 127.0.97.2:55567 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:18.894281 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:18.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:42575)
0929 00:55:18.887051 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:18.887067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:18.887212 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:18.887573 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:18.887577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:18.887918 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:18.888078 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:18.888086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.889136 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.889140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:18.890010 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:18.890019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.890666 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.890670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:18.890783 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:18.891312 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:18.891329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:18.894018 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":306,"threads_started":1}
W20260929 00:55:18.894454 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:18.886558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42575 (local address 127.0.97.62:46577)
0929 00:55:18.886763 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:18.886772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:18.887148 (+ 376us) server_negotiation.cc:407] Connection header received
0929 00:55:18.887357 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:18.887362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:18.887442 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:18.887528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:18.888189 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.889011 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.890123 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.890543 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.890632 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:18.894213 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42575: BlockingRecv error: recv got EOF from 127.0.97.3:42575 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:18 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:19.005115 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:18.996586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:50469)
0929 00:55:18.997158 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:18.997177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:18.997324 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:18.997658 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:18.997662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:18.998075 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:18.998248 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:18.998256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.999254 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:18.999259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.000151 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:19.000162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.000888 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.000894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.001078 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.001767 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:19.001784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:19.004878 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":251,"threads_started":1}
W20260929 00:55:19.005285 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:18.996675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50469 (local address 127.0.97.62:46577)
0929 00:55:18.996896 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:18.996905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:18.997261 (+ 356us) server_negotiation.cc:407] Connection header received
0929 00:55:18.997441 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:18.997446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:18.997514 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:18.997621 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:18.998387 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:18.999111 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.000284 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.000743 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.000883 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.005058 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50469: BlockingRecv error: recv got EOF from 127.0.97.1:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 00:55:19 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:19 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:19.462903 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:19.448221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:54091)
0929 00:55:19.448880 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:19.448899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:19.449008 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:19.449304 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:19.449308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:19.449698 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:19.449867 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:19.449874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.450816 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.450819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.451787 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:19.451798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.452625 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.452630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.452813 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.459876 (+ 7063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:19.459894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:19.462654 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":314,"threads_started":1}
W20260929 00:55:19.463094 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:19.448294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54091 (local address 127.0.97.62:46577)
0929 00:55:19.448472 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:19.448476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:19.448972 (+ 496us) server_negotiation.cc:407] Connection header received
0929 00:55:19.449119 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:19.449123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:19.449183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:19.449266 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:19.449980 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.450693 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.451899 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.452474 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.452625 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.462839 (+ 10214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54091: BlockingRecv error: recv got EOF from 127.0.97.2:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 00:55:19 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:19 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:19.903378 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:19.895079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:36289)
0929 00:55:19.895630 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:19.895646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:19.895759 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:19.896038 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:19.896041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:19.896443 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:19.896691 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:19.896700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.897624 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.897627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.898535 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:19.898545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.899207 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.899211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:19.899317 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.900055 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:19.900095 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:19.903132 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":250,"threads_started":1}
W20260929 00:55:19.903528 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:19.895192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36289 (local address 127.0.97.62:46577)
0929 00:55:19.895419 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:19.895426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:19.895725 (+ 299us) server_negotiation.cc:407] Connection header received
0929 00:55:19.895861 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:19.895864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:19.895924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:19.896001 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:19.896806 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.897511 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.898649 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:19.899068 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:19.899161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:19.903325 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36289: BlockingRecv error: recv got EOF from 127.0.97.3:36289 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:20.014038 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.006151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:59469)
0929 00:55:20.006722 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.006749 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.006904 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.007296 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.007301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.007818 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.007965 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.007973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.008937 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.008941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.009853 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.009863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.010509 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.010513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.010625 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.011193 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.011209 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.013828 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
W20260929 00:55:20.014266 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:20.006333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59469 (local address 127.0.97.62:46577)
0929 00:55:20.006496 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.006500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.006858 (+ 358us) server_negotiation.cc:407] Connection header received
0929 00:55:20.007038 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.007045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.007127 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.007219 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.008086 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.008820 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.009969 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.010372 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.010461 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.013999 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59469: BlockingRecv error: recv got EOF from 127.0.97.1:59469 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:20.472201 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.463773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:35111)
0929 00:55:20.464295 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.464317 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.464430 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.464738 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.464741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.465130 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.465290 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.465298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.466415 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.466419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.467320 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.467332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.467987 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.467991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.468104 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.468815 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.468832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.471867 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":293,"threads_started":1}
W20260929 00:55:20.472251 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:20.463877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35111 (local address 127.0.97.62:46577)
0929 00:55:20.464047 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.464051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.464391 (+ 340us) server_negotiation.cc:407] Connection header received
0929 00:55:20.464548 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.464551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.464611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.464698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.465410 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.466297 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.467441 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.467839 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.467927 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.472039 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35111: BlockingRecv error: recv got EOF from 127.0.97.2:35111 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:20 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:20.913336 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:20.904335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:49799)
0929 00:55:20.904825 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:20.904842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:20.904950 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:20.905231 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:20.905234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:20.905605 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:20.905809 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.905822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.907185 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.907189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.908253 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:20.908265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.909158 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.909162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:20.909280 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.909942 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:20.909964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:20.913005 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":288,"threads_started":1}
W20260929 00:55:20.913447 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:20.904373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49799 (local address 127.0.97.62:46577)
0929 00:55:20.904621 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:20.904630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:20.904919 (+ 289us) server_negotiation.cc:407] Connection header received
0929 00:55:20.905049 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:20.905053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:20.905112 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:20.905196 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:20.905953 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.907053 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.908387 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:20.908984 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:20.909088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:20.913186 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49799: BlockingRecv error: recv got EOF from 127.0.97.3:49799 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:21.023391 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.014995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:44915)
0929 00:55:21.015464 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.015485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.015594 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.015999 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.016004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.016489 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.016730 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.016741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.017799 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.017805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.018713 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.018724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.019448 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.019454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.019625 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.020175 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:21.020191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.023056 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
W20260929 00:55:21.023460 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:21.015141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44915 (local address 127.0.97.62:46577)
0929 00:55:21.015357 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.015363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.015561 (+ 198us) server_negotiation.cc:407] Connection header received
0929 00:55:21.015792 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.015795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.015855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.015960 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.016889 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.017656 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.018860 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.019295 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.019420 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.023229 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44915: BlockingRecv error: recv got EOF from 127.0.97.1:44915 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:21.481441 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.473253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:52061)
0929 00:55:21.473720 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.473739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.473851 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.474152 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.474155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.474539 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.474707 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.474715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.475645 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.475648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.476620 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.476631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.477307 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.477311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.477422 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.478034 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:21.478051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.481095 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
W20260929 00:55:21.481539 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:21.473327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52061 (local address 127.0.97.62:46577)
0929 00:55:21.473571 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.473577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.473815 (+ 238us) server_negotiation.cc:407] Connection header received
0929 00:55:21.473954 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.473958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.474015 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.474093 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.474817 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.475526 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.476734 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.477190 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.477286 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.481295 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52061: BlockingRecv error: recv got EOF from 127.0.97.2:52061 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:21 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:21.922681 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:21.914388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:57829)
0929 00:55:21.914836 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:21.914852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:21.914959 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:21.915243 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:21.915246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:21.915707 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:21.915889 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.915897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.916831 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.916835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.917742 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:21.917751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.918414 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.918419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:21.918583 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.919331 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:21.919348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:21.922445 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":264,"threads_started":1}
W20260929 00:55:21.922837 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:21.914438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57829 (local address 127.0.97.62:46577)
0929 00:55:21.914655 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:21.914663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:21.914931 (+ 268us) server_negotiation.cc:407] Connection header received
0929 00:55:21.915053 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:21.915056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:21.915115 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:21.915200 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:21.916006 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.916721 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.917868 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:21.918274 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:21.918363 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:21.922624 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57829: BlockingRecv error: recv got EOF from 127.0.97.3:57829 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:22.033934 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:22.024352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:42155)
0929 00:55:22.025315 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:22.025364 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:22.025506 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:22.025864 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:22.025869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:22.026361 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:22.026568 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.026578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.027586 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.027593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.028609 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.028620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.029370 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.029377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.029554 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.030174 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:22.030191 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:22.033564 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":755,"threads_started":1}
W20260929 00:55:22.034634 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:22.024468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42155 (local address 127.0.97.62:46577)
0929 00:55:22.024674 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:22.024682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:22.025458 (+ 776us) server_negotiation.cc:407] Connection header received
0929 00:55:22.025626 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:22.025630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:22.025707 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:22.025834 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:22.026698 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.027437 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.028738 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.029183 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.029339 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.033843 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42155: BlockingRecv error: recv got EOF from 127.0.97.1:42155 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:22.490451 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:22.482381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:47511)
0929 00:55:22.482865 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:22.482884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:22.483026 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:22.483380 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:22.483385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:22.483784 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:22.483920 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.483928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.485000 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.485008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.485954 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.485964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.486685 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.486691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.486866 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.487442 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:22.487458 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:22.490138 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
W20260929 00:55:22.490571 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:22.482460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47511 (local address 127.0.97.62:46577)
0929 00:55:22.482680 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:22.482689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:22.482970 (+ 281us) server_negotiation.cc:407] Connection header received
0929 00:55:22.483157 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:22.483162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:22.483246 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:22.483348 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:22.484068 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.484880 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.486090 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.486528 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.486660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.490336 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47511: BlockingRecv error: recv got EOF from 127.0.97.2:47511 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:22 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:22.931555 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:22.923532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:58977)
0929 00:55:22.924047 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:22.924062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:22.924222 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:22.924589 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:22.924594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:22.925008 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:22.925153 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.925161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.926170 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.926178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.927113 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:22.927124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.927845 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.927851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:22.928029 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.928659 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:22.928675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:22.931242 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
W20260929 00:55:22.931695 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:22.923716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58977 (local address 127.0.97.62:46577)
0929 00:55:22.923926 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:22.923931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:22.924138 (+ 207us) server_negotiation.cc:407] Connection header received
0929 00:55:22.924350 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:22.924356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:22.924432 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:22.924559 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:22.925310 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.926039 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.927254 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:22.927687 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:22.927828 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:22.931456 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58977: BlockingRecv error: recv got EOF from 127.0.97.3:58977 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:23.043670 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.035279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:39255)
0929 00:55:23.035858 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:23.035889 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:23.036032 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:23.036683 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:23.036687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:23.037148 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:23.037318 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.037326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.038303 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.038307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.039317 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.039328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.040126 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.040131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.040305 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.040833 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:23.040849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:23.043447 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":261,"threads_started":1}
W20260929 00:55:23.043841 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:23.035396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39255 (local address 127.0.97.62:46577)
0929 00:55:23.036345 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:23.036353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:23.036373 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:23.036422 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:23.036427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:23.036554 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:23.036636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:23.037442 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.038196 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.039434 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.039980 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.040113 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.043622 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39255: BlockingRecv error: recv got EOF from 127.0.97.1:39255 (error 108)
Metrics: {"server-negotiator.queue_time_us":778}
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:23.499612 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.491376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:52195)
0929 00:55:23.491841 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:23.491863 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:23.492015 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:23.492338 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:23.492343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:23.492918 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:23.493138 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.493147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.494184 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.494190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.495104 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.495113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.495809 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.495814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.495994 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.496599 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:23.496624 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:23.499288 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
W20260929 00:55:23.499712 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:23.491442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52195 (local address 127.0.97.62:46577)
0929 00:55:23.491659 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:23.491675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:23.491944 (+ 269us) server_negotiation.cc:407] Connection header received
0929 00:55:23.492131 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:23.492137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:23.492210 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:23.492309 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:23.493290 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.494025 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.495248 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.495653 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.495783 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.499475 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52195: BlockingRecv error: recv got EOF from 127.0.97.2:52195 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:23 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:23.940631 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:23.932740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:42579)
0929 00:55:23.933189 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:23.933205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:23.933331 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:23.933670 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:23.933674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:23.934059 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:23.934234 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.934243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.935209 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.935212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.936074 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:23.936083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.936737 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.936741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:23.936852 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.937396 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:23.937412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:23.940331 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
W20260929 00:55:23.940742 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:23.932888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42579 (local address 127.0.97.62:46577)
0929 00:55:23.933101 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:23.933107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:23.933307 (+ 200us) server_negotiation.cc:407] Connection header received
0929 00:55:23.933432 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:23.933435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:23.933505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:23.933585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:23.934361 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.935083 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.936187 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:23.936612 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:23.936696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:23.940486 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42579: BlockingRecv error: recv got EOF from 127.0.97.3:42579 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:24.052760 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.044834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:57441)
0929 00:55:24.045329 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.045355 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.045488 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.045865 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.045868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.046276 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.046421 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.046429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.047471 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.047475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.048373 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.048383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.049100 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.049104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.049231 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.049757 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:24.049773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.052510 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":293,"threads_started":1}
W20260929 00:55:24.052919 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:24.044886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57441 (local address 127.0.97.62:46577)
0929 00:55:24.045064 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.045071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.045439 (+ 368us) server_negotiation.cc:407] Connection header received
0929 00:55:24.045614 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.045619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.045709 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.045824 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.046537 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.047350 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.048536 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.048982 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.049087 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.052678 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57441: BlockingRecv error: recv got EOF from 127.0.97.1:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:24.513888 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.500706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:38653)
0929 00:55:24.501178 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.501199 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.501314 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.501609 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.501612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.502017 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.502186 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.502194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.503082 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.503085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.504014 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.504023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.504728 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.504732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.504868 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.510872 (+ 6004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:24.510891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.513633 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":291,"threads_started":1}
W20260929 00:55:24.514034 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:24.500755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38653 (local address 127.0.97.62:46577)
0929 00:55:24.500981 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.500991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.501273 (+ 282us) server_negotiation.cc:407] Connection header received
0929 00:55:24.501405 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.501409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.501466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.501563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.502295 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.502950 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.504126 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.504603 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.504702 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.513816 (+ 9114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38653: BlockingRecv error: recv got EOF from 127.0.97.2:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:24 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:24.949960 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:24.941691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:41363)
0929 00:55:24.942188 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:24.942204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:24.942332 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:24.942711 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:24.942714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:24.943047 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:24.943181 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.943188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.944539 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.944546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.945449 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:24.945459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.946073 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.946077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:24.946291 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.946898 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:24.946915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:24.949726 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":291,"threads_started":1}
W20260929 00:55:24.950106 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:24.941730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41363 (local address 127.0.97.62:46577)
0929 00:55:24.941909 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:24.941913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:24.942307 (+ 394us) server_negotiation.cc:407] Connection header received
0929 00:55:24.942456 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:24.942462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:24.942553 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:24.942634 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:24.943336 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.944373 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.945562 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:24.945958 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:24.946053 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:24.949900 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41363: BlockingRecv error: recv got EOF from 127.0.97.3:41363 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:25.061482 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.053545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:49299)
0929 00:55:25.054003 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.054021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.054128 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.054397 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.054399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.054768 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:25.054929 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.054938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.055826 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.055830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.056752 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.056762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.057435 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.057439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.057545 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.058174 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:25.058190 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:25.061150 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
W20260929 00:55:25.061561 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:25.053619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49299 (local address 127.0.97.62:46577)
0929 00:55:25.053861 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.053867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.054097 (+ 230us) server_negotiation.cc:407] Connection header received
0929 00:55:25.054205 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.054208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.054285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.054362 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.055040 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.055704 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.056870 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.057290 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.057378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.061332 (+ 3954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49299: BlockingRecv error: recv got EOF from 127.0.97.1:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:25.522728 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.514734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:35029)
0929 00:55:25.515258 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.515277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.515389 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.515663 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.515667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.516048 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:25.516223 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.516231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.517186 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.517189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.518124 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.518134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.518909 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.518913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.519049 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.519615 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:25.519632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:25.522447 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
W20260929 00:55:25.522923 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:25.514895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35029 (local address 127.0.97.62:46577)
0929 00:55:25.515051 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.515056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.515348 (+ 292us) server_negotiation.cc:407] Connection header received
0929 00:55:25.515481 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.515485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.515543 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.515621 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.516336 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.517071 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.518240 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.518655 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.518750 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.522685 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35029: BlockingRecv error: recv got EOF from 127.0.97.2:35029 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:25 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:25.958820 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:25.950825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:42423)
0929 00:55:25.951378 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:25.951393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:25.951534 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:25.951848 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:25.951853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:25.952236 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:25.952438 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.952448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.953455 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.953461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.954353 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:25.954363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.955063 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.955068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:25.955226 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.955820 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:25.955836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:25.958528 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":256,"threads_started":1}
W20260929 00:55:25.958982 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:25.950873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42423 (local address 127.0.97.62:46577)
0929 00:55:25.951080 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:25.951087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:25.951464 (+ 377us) server_negotiation.cc:407] Connection header received
0929 00:55:25.951651 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:25.951657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:25.951725 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:25.951820 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:25.952632 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.953327 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.954486 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:25.954898 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:25.955028 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:25.958742 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42423: BlockingRecv error: recv got EOF from 127.0.97.3:42423 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:26.070396 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:26.062494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:41315)
0929 00:55:26.062961 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:26.062981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:26.063089 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:26.063381 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:26.063384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:26.063732 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:26.063873 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.063881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.065231 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.065237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.066302 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.066314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.066907 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.066911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.067023 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.067595 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:26.067611 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:26.070153 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
W20260929 00:55:26.070549 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:26.062613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41315 (local address 127.0.97.62:46577)
0929 00:55:26.062961 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:26.062967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:26.063055 (+ 88us) server_negotiation.cc:407] Connection header received
0929 00:55:26.063181 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:26.063185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:26.063257 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:26.063340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:26.064012 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.065085 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.066420 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.066791 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.066881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.070334 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41315: BlockingRecv error: recv got EOF from 127.0.97.1:41315 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:26.532079 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:26.523786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:41193)
0929 00:55:26.524282 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:26.524305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:26.524420 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:26.524796 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:26.524800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:26.525146 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:26.525293 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.525300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.526284 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.526287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.527214 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.527224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.527884 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.527888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.528008 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.528725 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:26.528742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:26.531733 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
W20260929 00:55:26.532107 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:26.523917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41193 (local address 127.0.97.62:46577)
0929 00:55:26.524094 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:26.524099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:26.524382 (+ 283us) server_negotiation.cc:407] Connection header received
0929 00:55:26.524564 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:26.524569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:26.524641 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:26.524744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:26.525422 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.526147 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.527330 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.527747 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.527866 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.531913 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41193: BlockingRecv error: recv got EOF from 127.0.97.2:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:26 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:26.967123 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:26.959677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.3:38969)
0929 00:55:26.960175 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:26.960194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:26.960321 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:26.960616 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:26.960619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:26.961000 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:26.961144 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.961152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.962122 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.962126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.963022 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:26.963032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.963648 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.963652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:26.963762 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.964338 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:26.964354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:26.966879 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
W20260929 00:55:26.967289 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:26.959809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38969 (local address 127.0.97.62:46577)
0929 00:55:26.959967 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:26.959971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:26.960274 (+ 303us) server_negotiation.cc:407] Connection header received
0929 00:55:26.960420 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:26.960423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:26.960483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:26.960578 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:26.961251 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.961932 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.963131 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:26.963528 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:26.963623 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:26.967082 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38969: BlockingRecv error: recv got EOF from 127.0.97.3:38969 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 00:55:27 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:27 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:27.079815 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.071470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.1:50541)
0929 00:55:27.071963 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.071982 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.072122 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.072472 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.072476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.072903 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:27.073061 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.073069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.074086 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.074092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.075031 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.075042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.075769 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.075783 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.075980 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.076665 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:27.076689 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:27.079562 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":293,"threads_started":1}
W20260929 00:55:27.079944 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:27.071533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50541 (local address 127.0.97.62:46577)
0929 00:55:27.071752 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.071761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.072062 (+ 301us) server_negotiation.cc:407] Connection header received
0929 00:55:27.072250 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.072254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.072334 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.072434 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.073220 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.073942 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.075168 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.075622 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.075759 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.079739 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50541: BlockingRecv error: recv got EOF from 127.0.97.1:50541 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 00:55:27 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 00:55:27 dist-test-slave-pzsx krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 00:55:27.540755 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.533188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46577 (local address 127.0.97.2:56051)
0929 00:55:27.533733 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:27.533753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:27.533869 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:27.534144 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:27.534148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:27.534532 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:27.534696 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.534704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.535648 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.535651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.536588 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:27.536599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.537258 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.537261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:27.537378 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.537939 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:27.537956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:27.540444 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46577: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":319,"threads_started":1}
W20260929 00:55:27.540894 9335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:27.533258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56051 (local address 127.0.97.62:46577)
0929 00:55:27.533431 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.533435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.533829 (+ 394us) server_negotiation.cc:407] Connection header received
0929 00:55:27.533953 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.533957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.534019 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.534102 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.534817 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.535511 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.536702 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.537143 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.537234 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.540680 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56051: BlockingRecv error: recv got EOF from 127.0.97.2:56051 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:55:27.821516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9186
I20260929 00:55:27.833812 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.822694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.0.97.1:41909)
0929 00:55:27.823203 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.823210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.823248 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:55:27.823309 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.823315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.823554 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.823706 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.825283 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.826355 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.827508 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.828051 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.828149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.830646 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.830653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.830669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.830705 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.833304 (+ 2599us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 00:55:27.833572 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54538: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
W20260929 00:55:27.833952 9646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54538: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 00:55:27.834038 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9186: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41909: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 00:55:27.834276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9186
I20260929 00:55:27.851290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9337
I20260929 00:55:27.862716 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.852270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.97.2:46569)
0929 00:55:27.852858 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.852865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.852896 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:55:27.852956 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.852962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.853187 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.853352 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.854530 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.855610 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.856816 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.857375 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.857473 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.859818 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.859827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.859841 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.859876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.862205 (+ 2329us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 00:55:27.862479 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58800: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
W20260929 00:55:27.862854 9647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58800: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 00:55:27.862869 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9337: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46569: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 00:55:27.863023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9337
I20260929 00:55:27.880069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9468
I20260929 00:55:27.891878 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:27.881143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.97.3:34297)
0929 00:55:27.881760 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:27.881767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:27.881795 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:55:27.881856 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:27.881862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:27.882086 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:27.882246 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:27.883485 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.884562 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.885685 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:27.886246 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:27.886345 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:27.888780 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:27.888789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:27.888804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:27.888836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:27.891358 (+ 2522us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 00:55:27.891604 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56240: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":323,"threads_started":1}
W20260929 00:55:27.891987 9648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56240: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 00:55:27.892030 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9468: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34297: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 00:55:27.892231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9468
I20260929 00:55:27.908478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9117
I20260929 00:55:27.909788 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:28.024489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9117
2026-09-29T00:55:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20985 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T00:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:28Z Disabled control of system clock
I20260929 00:55:28.105126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35031
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 00:55:28.433017 9653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.433314 9653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.433391 9653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.440584 9653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:28.440692 9653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.440730 9653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.440793 9653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:28.440836 9653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:28.450083 9653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:28.451357 9653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.453100 9653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:28.465101 9658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.465071 9659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:28.465340 9661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:28.466049 9653 server_base.cc:1034] running on GCE node
I20260929 00:55:28.466840 9653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:28.468250 9653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:28.469470 9653 hybrid_clock.cc:648] HybridClock initialized: now 1790643328469400 us; error 63 us; skew 500 ppm
I20260929 00:55:28.469916 9653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:28.472229 9653 webserver.cc:460] Webserver started at http://127.0.97.62:42883/ using document root <none> and password file <none>
I20260929 00:55:28.472960 9653 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:28.473089 9653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:28.473404 9653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:28.476219 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "d38ad1cd58e44fd5aa6482e329d5962f"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-pzsx"
server_key: "2e7644cfe39ed46cb0eb991d0af5dae3"
server_key_iv: "e0fd23b6088bdda1655d48dd2f373826"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.477028 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "d38ad1cd58e44fd5aa6482e329d5962f"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-pzsx"
server_key: "2e7644cfe39ed46cb0eb991d0af5dae3"
server_key_iv: "e0fd23b6088bdda1655d48dd2f373826"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.481848 9653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 00:55:28.485276 9667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:28.486547 9653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:28.486726 9653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "d38ad1cd58e44fd5aa6482e329d5962f"
format_stamp: "Formatted at 2026-09-29 00:55:28 on dist-test-slave-pzsx"
server_key: "2e7644cfe39ed46cb0eb991d0af5dae3"
server_key_iv: "e0fd23b6088bdda1655d48dd2f373826"
server_key_version: "encryptionkey@0"
I20260929 00:55:28.486937 9653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:28.506568 9653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:28.507418 9653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:28.507725 9653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:28.525203 9653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35031
I20260929 00:55:28.525207 9718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35031 every 8 connection(s)
I20260929 00:55:28.526548 9653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:28.530177 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:28.536295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9653
I20260929 00:55:28.536574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:55:28.537194 388 external_mini_cluster.cc:1428] Setting key 045c6ee5c9b4fe469ac1b33720dff0c9
I20260929 00:55:28.541031 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f: Bootstrap starting.
I20260929 00:55:28.544463 9719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:28.545684 9719 log.cc:824] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:28.548907 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f: No bootstrap required, opened a new log
I20260929 00:55:28.551234 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:28.539335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.97.62:35031)
0929 00:55:28.540351 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:28.540370 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:28.540412 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:55:28.542153 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:28.542184 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:28.542673 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:28.543006 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:28.543699 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.545074 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.546810 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:28.547519 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:28.547644 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:28.548855 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:28.548862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:28.548876 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:28.549551 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:28.549630 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:28.549650 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:28.549737 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:28.549967 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 00:55:28.550303 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":362,"threads_started":1}
I20260929 00:55:28.557854 9719 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } }
I20260929 00:55:28.558362 9719 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:28.558521 9719 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38ad1cd58e44fd5aa6482e329d5962f, State: Initialized, Role: FOLLOWER
I20260929 00:55:28.559226 9719 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } }
I20260929 00:55:28.559514 9719 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:28.559659 9719 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:28.559857 9719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:28.562984 9719 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } }
I20260929 00:55:28.563495 9719 leader_election.cc:302] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38ad1cd58e44fd5aa6482e329d5962f; no voters:
I20260929 00:55:28.564028 9719 leader_election.cc:288] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:28.564195 9724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:28.565594 9724 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [term 1 LEADER]: Becoming Leader. State: Replica: d38ad1cd58e44fd5aa6482e329d5962f, State: Running, Role: LEADER
I20260929 00:55:28.566097 9724 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } }
I20260929 00:55:28.566458 9719 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:28.568905 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38ad1cd58e44fd5aa6482e329d5962f. Latest consensus state: current_term: 1 leader_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } } }
I20260929 00:55:28.569247 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:28.569712 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38ad1cd58e44fd5aa6482e329d5962f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35031 } } }
I20260929 00:55:28.570024 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:28.570156 9733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:28.575558 9733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:28.584697 9733 catalog_manager.cc:1347] Generated new cluster ID: 1ecc274dc4ee4663ac90f775524cd2c5
I20260929 00:55:28.584810 9733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:28.627791 9733 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:28.629217 9733 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:28.640640 9733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38ad1cd58e44fd5aa6482e329d5962f: Generated new TSK 0
I20260929 00:55:28.641498 9733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:28.652818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--builtin_ntp_servers=127.0.97.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 00:55:28.977721 9743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:28.977994 9743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:28.978153 9743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:28.985466 9743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:28.985561 9743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:28.985760 9743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:28.994879 9743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:28.996203 9743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:28.998113 9743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:29.011047 9748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.011076 9749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.011274 9751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:29.011875 9743 server_base.cc:1034] running on GCE node
I20260929 00:55:29.012454 9743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:29.013399 9743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:29.014698 9743 hybrid_clock.cc:648] HybridClock initialized: now 1790643329014642 us; error 42 us; skew 500 ppm
I20260929 00:55:29.015058 9743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:29.017241 9743 webserver.cc:460] Webserver started at http://127.0.97.1:38023/ using document root <none> and password file <none>
I20260929 00:55:29.017912 9743 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:29.018030 9743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:29.018352 9743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:29.021159 9743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "b7084a4bee7a41c6923f72d98f93fd84"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "31d35a29c8a3a731009dd0ef7bb653ab"
server_key_iv: "59590b1e6e30a828200a03f5ed905960"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.021934 9743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b7084a4bee7a41c6923f72d98f93fd84"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "31d35a29c8a3a731009dd0ef7bb653ab"
server_key_iv: "59590b1e6e30a828200a03f5ed905960"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.026880 9743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:55:29.030143 9757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.031461 9743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:55:29.031630 9743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "b7084a4bee7a41c6923f72d98f93fd84"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "31d35a29c8a3a731009dd0ef7bb653ab"
server_key_iv: "59590b1e6e30a828200a03f5ed905960"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.031837 9743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:29.048372 9743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:29.049237 9743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:29.049507 9743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:29.050495 9743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:29.051932 9743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:29.052016 9743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.052093 9743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:29.052146 9743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.084532 9743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45301
I20260929 00:55:29.084590 9869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45301 every 8 connection(s)
I20260929 00:55:29.085826 9743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:29.087001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9743
I20260929 00:55:29.087180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:29.088020 388 external_mini_cluster.cc:1428] Setting key 1bf97003e2898d1b2ab7fac5519c7981
I20260929 00:55:29.091187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--builtin_ntp_servers=127.0.97.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:55:29.100865 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.088616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42977 (local address 127.0.97.62:35031)
0929 00:55:29.089117 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.089121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.090671 (+ 1550us) server_negotiation.cc:407] Connection header received
0929 00:55:29.093422 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.093426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.093487 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.093580 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:29.094915 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.095635 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.098120 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.098490 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.098583 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.100116 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:29.100120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:29.100123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:29.100139 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:29.100178 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:29.100190 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:29.100256 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.100539 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.100622 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260929 00:55:29.102248 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.089358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35031 (local address 127.0.97.1:42977)
0929 00:55:29.090515 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.090557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:29.093144 (+ 2587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:29.093836 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:29.093849 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:29.093961 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:29.094741 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.094765 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.095778 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.095786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.097965 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.097987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.098713 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.098720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.099027 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.099783 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:29.099788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:29.099837 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:29.099931 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:29.100283 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:29.100296 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:29.100306 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:55:29.100484 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 00:55:29.100922 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260929 00:55:29.103889 9870 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35031
I20260929 00:55:29.104190 9870 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:29.105106 9870 heartbeater.cc:511] Master 127.0.97.62:35031 requested a full tablet report, sending...
I20260929 00:55:29.107363 9684 ts_manager.cc:194] Registered new tserver with Master: b7084a4bee7a41c6923f72d98f93fd84 (127.0.97.1:45301)
I20260929 00:55:29.109730 9684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42977
W20260929 00:55:29.431697 9874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:29.431993 9874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:29.432137 9874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:29.439714 9874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:29.439824 9874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:29.440001 9874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:29.449223 9874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:29.450515 9874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:29.452370 9874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:29.464939 9882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.464918 9880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.465096 9879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:29.469964 9874 server_base.cc:1034] running on GCE node
I20260929 00:55:29.470522 9874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:29.471406 9874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:29.472599 9874 hybrid_clock.cc:648] HybridClock initialized: now 1790643329472527 us; error 59 us; skew 500 ppm
I20260929 00:55:29.472922 9874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:29.475370 9874 webserver.cc:460] Webserver started at http://127.0.97.2:46487/ using document root <none> and password file <none>
I20260929 00:55:29.476051 9874 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:29.476169 9874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:29.476550 9874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:29.479357 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "3eb5297225ae4d3ebec4122535c2c14a"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "dd53f614679b27353ed3f985959008b6"
server_key_iv: "2f9c4bea13484c6b341103a6b50e93b0"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.480095 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3eb5297225ae4d3ebec4122535c2c14a"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "dd53f614679b27353ed3f985959008b6"
server_key_iv: "2f9c4bea13484c6b341103a6b50e93b0"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.485219 9874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 00:55:29.488483 9888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.489934 9874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:55:29.490092 9874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "3eb5297225ae4d3ebec4122535c2c14a"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "dd53f614679b27353ed3f985959008b6"
server_key_iv: "2f9c4bea13484c6b341103a6b50e93b0"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.490317 9874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:29.503952 9874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:29.504837 9874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:29.505064 9874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:29.505924 9874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:29.507354 9874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:29.507436 9874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.507519 9874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:29.507573 9874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.543810 9874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46137
I20260929 00:55:29.543884 10000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46137 every 8 connection(s)
I20260929 00:55:29.545135 9874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:29.548079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 9874
I20260929 00:55:29.548233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:29.548713 388 external_mini_cluster.cc:1428] Setting key f779dc3e4db10d1f14f9d3afbfba229c
I20260929 00:55:29.551368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--builtin_ntp_servers=127.0.97.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 00:55:29.562747 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.547570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58763 (local address 127.0.97.62:35031)
0929 00:55:29.547799 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.547805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.549420 (+ 1615us) server_negotiation.cc:407] Connection header received
0929 00:55:29.551142 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.551150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.551233 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.551348 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:29.552472 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.553638 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.555129 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.555825 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.555966 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.561842 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:29.561851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:29.561858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:29.561884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:29.561928 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:29.561946 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:29.562043 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.562247 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.562355 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:55:29.563597 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.548160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35031 (local address 127.0.97.2:58763)
0929 00:55:29.549266 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.549319 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:29.550895 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:29.551688 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:29.551696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:29.551777 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:29.552310 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.552325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.553772 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.553775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.554990 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.555001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.556017 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.556021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.556212 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.561510 (+ 5298us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:29.561513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:29.561556 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:29.561632 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:29.562045 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:29.562052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:29.562058 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:55:29.562214 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 00:55:29.562507 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":365,"threads_started":1}
I20260929 00:55:29.565034 10001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35031
I20260929 00:55:29.565336 10001 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:29.566054 10001 heartbeater.cc:511] Master 127.0.97.62:35031 requested a full tablet report, sending...
I20260929 00:55:29.567463 9684 ts_manager.cc:194] Registered new tserver with Master: 3eb5297225ae4d3ebec4122535c2c14a (127.0.97.2:46137)
I20260929 00:55:29.568218 9684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58763
W20260929 00:55:29.876605 10004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:29.876886 10004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:29.877059 10004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:29.884205 10004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:29.884311 10004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:29.884486 10004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:29.893741 10004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:29.895164 10004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:29.896921 10004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:29.909335 10009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.909256 10010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:29.910382 10012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:29.911201 10004 server_base.cc:1034] running on GCE node
I20260929 00:55:29.911739 10004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:29.912667 10004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:29.913889 10004 hybrid_clock.cc:648] HybridClock initialized: now 1790643329913831 us; error 65 us; skew 500 ppm
I20260929 00:55:29.914242 10004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:29.916417 10004 webserver.cc:460] Webserver started at http://127.0.97.3:33185/ using document root <none> and password file <none>
I20260929 00:55:29.917143 10004 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:29.917277 10004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:29.917598 10004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:29.920416 10004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "70bf97e18ec340d78293abdc874898ed"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "933df86051674ed72aa25538a84e963f"
server_key_iv: "f5fd737bcc542c9abc237667c38a15e6"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.921290 10004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "70bf97e18ec340d78293abdc874898ed"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "933df86051674ed72aa25538a84e963f"
server_key_iv: "f5fd737bcc542c9abc237667c38a15e6"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.926376 10004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:55:29.929705 10018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.930972 10004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:29.931161 10004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "70bf97e18ec340d78293abdc874898ed"
format_stamp: "Formatted at 2026-09-29 00:55:29 on dist-test-slave-pzsx"
server_key: "933df86051674ed72aa25538a84e963f"
server_key_iv: "f5fd737bcc542c9abc237667c38a15e6"
server_key_version: "encryptionkey@0"
I20260929 00:55:29.931380 10004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:29.947328 10004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:29.948166 10004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:29.948455 10004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:29.949404 10004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:29.950826 10004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:29.950944 10004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.951097 10004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:29.951187 10004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:29.983398 10004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33307
I20260929 00:55:29.983490 10130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33307 every 8 connection(s)
I20260929 00:55:29.984768 10004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:29.994004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10004
I20260929 00:55:29.994171 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:29.994755 388 external_mini_cluster.cc:1428] Setting key b917d24a7b4d64fd00887f128264bc15
I20260929 00:55:29.997117 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.987176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59881 (local address 127.0.97.62:35031)
0929 00:55:29.987382 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:29.987388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:29.988883 (+ 1495us) server_negotiation.cc:407] Connection header received
0929 00:55:29.990660 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:29.990663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:29.990733 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:29.990816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:29.991897 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.992639 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.994104 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.994749 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.994874 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.996345 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:29.996351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:29.996356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:29.996376 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:29.996416 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:29.996434 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:29.996565 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:29.996780 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 00:55:29.996857 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:55:29.998273 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:29.987758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35031 (local address 127.0.97.3:59881)
0929 00:55:29.988700 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:29.988757 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:29.990395 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:29.991005 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:29.991013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:29.991096 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:29.991761 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.991779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.992761 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.992765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.993964 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:29.993975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:29.994979 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:29.995006 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:29.995254 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:29.996005 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:29.996008 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:29.996058 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:29.996149 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:29.996566 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:29.996579 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:29.996589 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:55:29.996746 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 00:55:29.997156 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260929 00:55:29.999387 10131 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35031
I20260929 00:55:29.999658 10131 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:30.000360 10131 heartbeater.cc:511] Master 127.0.97.62:35031 requested a full tablet report, sending...
I20260929 00:55:30.001605 9684 ts_manager.cc:194] Registered new tserver with Master: 70bf97e18ec340d78293abdc874898ed (127.0.97.3:33307)
I20260929 00:55:30.002328 9684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59881
I20260929 00:55:30.009811 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:55:30.015725 9872 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:30.014136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.97.62:35031)
0929 00:55:30.014308 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.014312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.014854 (+ 542us) server_negotiation.cc:407] Connection header received
0929 00:55:30.014980 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.014983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.015053 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.015130 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.015468 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56262: BlockingRecv error: recv got EOF from 127.0.0.1:56262 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 00:55:30.016229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9743
I20260929 00:55:30.023340 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.017166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.0.97.1:45301)
0929 00:55:30.017675 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.017689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.017713 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:55:30.017769 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.017775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.017976 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.018092 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.018395 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.019338 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.020454 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.021109 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.021209 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:30.022082 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.022090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.022120 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:30.022679 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.022721 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:30.022738 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.022819 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.022940 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.023049 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":269,"threads_started":1}
I20260929 00:55:30.024474 9865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:30.108858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9743
I20260929 00:55:30.126307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9874
I20260929 00:55:30.133719 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.127179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.0.97.2:46137)
0929 00:55:30.127738 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.127745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.127771 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:30.127837 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.127844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.128061 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.128195 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.128566 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.129570 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.130725 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.131289 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.131384 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:30.132366 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.132373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.132386 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:30.133036 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.133077 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:30.133099 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.133184 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.133312 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.133428 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":298,"threads_started":1}
I20260929 00:55:30.134949 9996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:30.216102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9874
I20260929 00:55:30.232389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10004
I20260929 00:55:30.240605 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.233433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.0.97.3:33307)
0929 00:55:30.234053 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.234060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.234090 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:55:30.234152 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.234158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.234381 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.234578 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.235032 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.236055 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.237369 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:30.237914 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:30.238052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:30.239134 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.239143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.239156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:30.239727 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.239775 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:30.239792 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.239915 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.240060 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.240210 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":305,"threads_started":1}
I20260929 00:55:30.241809 10126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:30.323490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10004
I20260929 00:55:30.339036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 9653
I20260929 00:55:30.340246 9714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:30.440619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 9653
2026-09-29T00:55:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2406 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T00:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:30Z Disabled control of system clock
I20260929 00:55:30.511795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42725
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:55:30.831099 10150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:30.831377 10150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:30.831460 10150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:30.838366 10150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:30.838475 10150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:30.838531 10150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:30.838588 10150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:30.838625 10150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:30.847383 10150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:30.848592 10150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:30.850286 10150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:30.861521 10155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:30.861604 10158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:30.861562 10156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:30.862439 10150 server_base.cc:1034] running on GCE node
I20260929 00:55:30.863219 10150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:30.864600 10150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:30.865825 10150 hybrid_clock.cc:648] HybridClock initialized: now 1790643330865788 us; error 49 us; skew 500 ppm
I20260929 00:55:30.866190 10150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:30.868288 10150 webserver.cc:460] Webserver started at http://127.0.97.62:42757/ using document root <none> and password file <none>
I20260929 00:55:30.869081 10150 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:30.869210 10150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:30.869532 10150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:30.872385 10150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "e83567e1635b4a1e9fec02da3a1663a1"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-pzsx"
server_key: "0e6bf00663092247f077ecbc7681db3a"
server_key_iv: "415b7c0bf6fd83426f08e3157b9f7144"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.873237 10150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "e83567e1635b4a1e9fec02da3a1663a1"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-pzsx"
server_key: "0e6bf00663092247f077ecbc7681db3a"
server_key_iv: "415b7c0bf6fd83426f08e3157b9f7144"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.877995 10150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 00:55:30.881237 10164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:30.882547 10150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:30.882735 10150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "e83567e1635b4a1e9fec02da3a1663a1"
format_stamp: "Formatted at 2026-09-29 00:55:30 on dist-test-slave-pzsx"
server_key: "0e6bf00663092247f077ecbc7681db3a"
server_key_iv: "415b7c0bf6fd83426f08e3157b9f7144"
server_key_version: "encryptionkey@0"
I20260929 00:55:30.882948 10150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:30.888535 10150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:30.889278 10150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:30.889567 10150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:30.907181 10150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42725
I20260929 00:55:30.907222 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42725 every 8 connection(s)
I20260929 00:55:30.908439 10150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:30.911926 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:30.912003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10150
I20260929 00:55:30.912230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:55:30.912885 388 external_mini_cluster.cc:1428] Setting key 2441da2c4923086dda5dc6965cabf110
I20260929 00:55:30.921093 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:30.914803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.97.62:42725)
0929 00:55:30.915827 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:30.915837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:30.915885 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:55:30.917677 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:30.917698 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:30.918114 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:30.918445 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:30.919511 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:30.919517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:30.919529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:30.919542 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:30.919602 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:30.919617 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:30.919779 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:30.919897 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 00:55:30.920162 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":410,"threads_started":1}
I20260929 00:55:30.924070 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1: Bootstrap starting.
I20260929 00:55:30.926904 10216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:30.927850 10216 log.cc:824] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:30.930603 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1: No bootstrap required, opened a new log
I20260929 00:55:30.937561 10216 raft_consensus.cc:348] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } }
I20260929 00:55:30.937898 10216 raft_consensus.cc:374] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:30.937994 10216 raft_consensus.cc:729] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83567e1635b4a1e9fec02da3a1663a1, State: Initialized, Role: FOLLOWER
I20260929 00:55:30.938513 10216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } }
I20260929 00:55:30.938771 10216 raft_consensus.cc:388] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:30.938928 10216 raft_consensus.cc:482] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:30.939113 10216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:30.941238 10216 raft_consensus.cc:504] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } }
I20260929 00:55:30.941711 10216 leader_election.cc:302] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e83567e1635b4a1e9fec02da3a1663a1; no voters:
I20260929 00:55:30.942273 10216 leader_election.cc:288] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:30.942404 10221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:30.943102 10221 raft_consensus.cc:686] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [term 1 LEADER]: Becoming Leader. State: Replica: e83567e1635b4a1e9fec02da3a1663a1, State: Running, Role: LEADER
I20260929 00:55:30.943590 10221 consensus_queue.cc:237] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } }
I20260929 00:55:30.944271 10216 sys_catalog.cc:564] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:30.946388 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e83567e1635b4a1e9fec02da3a1663a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } } }
I20260929 00:55:30.946364 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e83567e1635b4a1e9fec02da3a1663a1. Latest consensus state: current_term: 1 leader_uuid: "e83567e1635b4a1e9fec02da3a1663a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83567e1635b4a1e9fec02da3a1663a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42725 } } }
I20260929 00:55:30.946898 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:30.949292 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:30.949528 10226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:30.954670 10226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:30.963438 10226 catalog_manager.cc:1347] Generated new cluster ID: 764e667722b64445b2726846dd1a6488
I20260929 00:55:30.963562 10226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:30.979034 10226 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:30.979418 10226 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:30.985623 10226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e83567e1635b4a1e9fec02da3a1663a1: Generated new TSK 0
I20260929 00:55:30.986409 10226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:55:30.993090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--builtin_ntp_servers=127.0.97.20:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 00:55:31.340094 10240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:31.340375 10240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:31.340464 10240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:31.347723 10240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:31.347837 10240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:31.347947 10240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:31.357141 10240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:31.358359 10240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:31.360051 10240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:31.372880 10245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:31.372902 10246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:31.373112 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:31.373823 10240 server_base.cc:1034] running on GCE node
I20260929 00:55:31.374347 10240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:31.375167 10240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:31.376359 10240 hybrid_clock.cc:648] HybridClock initialized: now 1790643331376313 us; error 44 us; skew 500 ppm
I20260929 00:55:31.376698 10240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:31.379127 10240 webserver.cc:460] Webserver started at http://127.0.97.1:36841/ using document root <none> and password file <none>
I20260929 00:55:31.379860 10240 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:31.379984 10240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:31.380293 10240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:31.385334 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "6bc8b65a0c084cc49d0ffb833d204d05"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "02f46ef50990740331a3e33419164e7b"
server_key_iv: "65162a06f96ebf3f7e52f6fa2ad17b10"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.386197 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc8b65a0c084cc49d0ffb833d204d05"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "02f46ef50990740331a3e33419164e7b"
server_key_iv: "65162a06f96ebf3f7e52f6fa2ad17b10"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.391156 10240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 00:55:31.394917 10254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.396212 10240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:31.396394 10240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "6bc8b65a0c084cc49d0ffb833d204d05"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "02f46ef50990740331a3e33419164e7b"
server_key_iv: "65162a06f96ebf3f7e52f6fa2ad17b10"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.396720 10240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:31.401983 10240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:31.402729 10240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:31.402985 10240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:31.403829 10240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:31.405268 10240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:31.405385 10240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.405514 10240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:31.405601 10240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.437780 10240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43871
I20260929 00:55:31.437808 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43871 every 8 connection(s)
I20260929 00:55:31.439108 10240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:31.447132 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:31.441673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50905 (local address 127.0.97.62:42725)
0929 00:55:31.442113 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:31.442120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:31.443355 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 00:55:31.445118 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:31.445125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:31.445193 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:31.445288 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:31.446560 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:31.446563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:31.446566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:31.446580 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:31.446616 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:31.446627 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:31.446718 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:31.446875 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 00:55:31.446919 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260929 00:55:31.448244 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10240
I20260929 00:55:31.448242 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:31.442305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42725 (local address 127.0.97.1:50905)
0929 00:55:31.443197 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:31.443239 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:31.444864 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:31.445528 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:31.445537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:31.445628 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:31.446296 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:31.446298 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:31.446335 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:31.446417 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:31.446709 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:31.446715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:31.446721 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:55:31.446921 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 00:55:31.447134 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":280,"threads_started":1}
I20260929 00:55:31.448448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:31.449246 388 external_mini_cluster.cc:1428] Setting key 28de44df23ba5e291b89c91e333c6451
I20260929 00:55:31.449860 10367 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42725
I20260929 00:55:31.450157 10367 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:31.450737 10367 heartbeater.cc:511] Master 127.0.97.62:42725 requested a full tablet report, sending...
I20260929 00:55:31.452064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--builtin_ntp_servers=127.0.97.20:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 00:55:31.452891 10181 ts_manager.cc:194] Registered new tserver with Master: 6bc8b65a0c084cc49d0ffb833d204d05 (127.0.97.1:43871)
W20260929 00:55:31.770970 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:31.771226 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:31.771312 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:31.778340 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:31.778442 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:31.778548 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:31.787762 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:31.788950 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:31.790632 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:31.803363 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:31.803429 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:31.803658 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:31.804443 10371 server_base.cc:1034] running on GCE node
I20260929 00:55:31.805073 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:31.805962 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:31.807148 10371 hybrid_clock.cc:648] HybridClock initialized: now 1790643331807109 us; error 29 us; skew 500 ppm
I20260929 00:55:31.807459 10371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:31.809636 10371 webserver.cc:460] Webserver started at http://127.0.97.2:40537/ using document root <none> and password file <none>
I20260929 00:55:31.810321 10371 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:31.810429 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:31.810767 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:31.813650 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "c442ecd2abea46eaa72a43c0f18af21d"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "a60a8323830f9a5bfa8f0e35c7dc05b0"
server_key_iv: "3d1531697eb38640d5b7a2f198b9170d"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.814417 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c442ecd2abea46eaa72a43c0f18af21d"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "a60a8323830f9a5bfa8f0e35c7dc05b0"
server_key_iv: "3d1531697eb38640d5b7a2f198b9170d"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.819262 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:55:31.822626 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.823868 10371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:55:31.824041 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "c442ecd2abea46eaa72a43c0f18af21d"
format_stamp: "Formatted at 2026-09-29 00:55:31 on dist-test-slave-pzsx"
server_key: "a60a8323830f9a5bfa8f0e35c7dc05b0"
server_key_iv: "3d1531697eb38640d5b7a2f198b9170d"
server_key_version: "encryptionkey@0"
I20260929 00:55:31.824266 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:31.829744 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:31.830482 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:31.830746 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:31.831604 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:31.833042 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:31.833159 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.833295 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:31.833384 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:31.865410 10371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34411
I20260929 00:55:31.865466 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34411 every 8 connection(s)
I20260929 00:55:31.866750 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:31.873026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10371
I20260929 00:55:31.873209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:31.873709 388 external_mini_cluster.cc:1428] Setting key 8c20a909a925b071d0a5241fedf62f9a
I20260929 00:55:31.875509 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:31.869328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54937 (local address 127.0.97.62:42725)
0929 00:55:31.869532 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:31.869537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:31.870892 (+ 1355us) server_negotiation.cc:407] Connection header received
0929 00:55:31.872547 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:31.872556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:31.872624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:31.872714 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:31.874747 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:31.874753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:31.874757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:31.874776 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:31.874812 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:31.874826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:31.874935 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:31.875112 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 00:55:31.875164 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:55:31.876362 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:31.869860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42725 (local address 127.0.97.2:54937)
0929 00:55:31.870739 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:31.870780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:31.872241 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:31.873507 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:31.873520 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:31.873643 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:31.874374 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:31.874376 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:31.874414 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:31.874538 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:31.874923 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:31.874930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:31.874936 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:55:31.875147 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 00:55:31.875348 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260929 00:55:31.876607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--builtin_ntp_servers=127.0.97.20:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 00:55:31.877755 10498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42725
I20260929 00:55:31.878073 10498 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:31.878628 10498 heartbeater.cc:511] Master 127.0.97.62:42725 requested a full tablet report, sending...
I20260929 00:55:31.879933 10181 ts_manager.cc:194] Registered new tserver with Master: c442ecd2abea46eaa72a43c0f18af21d (127.0.97.2:34411)
W20260929 00:55:32.201807 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:32.202075 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:32.202241 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:32.209425 10501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:32.209524 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:32.209712 10501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:32.218881 10501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:32.220176 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:32.221925 10501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:32.235157 10506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:32.235112 10509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:32.235213 10507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:32.237913 10501 server_base.cc:1034] running on GCE node
I20260929 00:55:32.238504 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:32.239454 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:32.240677 10501 hybrid_clock.cc:648] HybridClock initialized: now 1790643332240645 us; error 63 us; skew 500 ppm
I20260929 00:55:32.240993 10501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:55:32.243126 10501 webserver.cc:460] Webserver started at http://127.0.97.3:43685/ using document root <none> and password file <none>
I20260929 00:55:32.243811 10501 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:32.243940 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:32.244258 10501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:32.247169 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "a4024bd85c8246f2a60a707b542a048b"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-pzsx"
server_key: "7019308496f54798953e8fd9c65681a5"
server_key_iv: "506220ed75f376d720861d0d50930c0f"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.247937 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4024bd85c8246f2a60a707b542a048b"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-pzsx"
server_key: "7019308496f54798953e8fd9c65681a5"
server_key_iv: "506220ed75f376d720861d0d50930c0f"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.253264 10501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 00:55:32.256597 10515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:32.258072 10501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:55:32.258240 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "a4024bd85c8246f2a60a707b542a048b"
format_stamp: "Formatted at 2026-09-29 00:55:32 on dist-test-slave-pzsx"
server_key: "7019308496f54798953e8fd9c65681a5"
server_key_iv: "506220ed75f376d720861d0d50930c0f"
server_key_version: "encryptionkey@0"
I20260929 00:55:32.258487 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:32.263996 10501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:32.264815 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:32.265066 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:32.265918 10501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:32.267215 10501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:32.267345 10501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:32.267465 10501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:32.267552 10501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:32.298981 10501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44849
I20260929 00:55:32.299031 10627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44849 every 8 connection(s)
I20260929 00:55:32.300307 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:32.308178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10501
I20260929 00:55:32.308233 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.302882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44291 (local address 127.0.97.62:42725)
0929 00:55:32.303052 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.303056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.304382 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 00:55:32.306070 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.306075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.306150 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.306270 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.307495 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.307501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.307505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.307531 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.307561 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.307572 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.307705 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.307890 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.307958 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:55:32.308403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:32.308835 388 external_mini_cluster.cc:1428] Setting key 5a331aaebcdf6db2bf14a5f3ec7cab8f
I20260929 00:55:32.309211 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.303338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42725 (local address 127.0.97.3:44291)
0929 00:55:32.304220 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:32.304255 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:32.305802 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:32.306495 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:32.306503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:32.306590 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:32.307203 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:55:32.307205 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:32.307246 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:55:32.307325 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:32.307664 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:32.307674 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:32.307683 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:55:32.307966 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 00:55:32.308212 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":278,"threads_started":1}
I20260929 00:55:32.310532 10628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42725
I20260929 00:55:32.310879 10628 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:32.311563 10628 heartbeater.cc:511] Master 127.0.97.62:42725 requested a full tablet report, sending...
I20260929 00:55:32.312954 10180 ts_manager.cc:194] Registered new tserver with Master: a4024bd85c8246f2a60a707b542a048b (127.0.97.3:44849)
I20260929 00:55:32.324172 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 00:55:32.330291 10369 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:55:32.328757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.0.97.62:42725)
0929 00:55:32.328926 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.328932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.329422 (+ 490us) server_negotiation.cc:407] Connection header received
0929 00:55:32.329613 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.329618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.329676 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.329778 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.330082 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44086: BlockingRecv error: recv got EOF from 127.0.0.1:44086 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 00:55:32.331203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10240
I20260929 00:55:32.334389 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.332077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.97.1:43871)
0929 00:55:32.332594 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.332601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.332626 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:55:32.332683 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.332688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.332901 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.333023 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.333839 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.333846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.333857 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.333870 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.333915 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.333932 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.334036 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.334085 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.334162 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260929 00:55:32.335510 10362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:32.416666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10240
I20260929 00:55:32.433157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10371
I20260929 00:55:32.436443 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.434022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.0.97.2:34411)
0929 00:55:32.434576 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.434583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.434609 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:55:32.434669 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.434676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.434895 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.435021 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.435844 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.435851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.435863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.435881 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.435923 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.435937 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.436039 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.436120 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.436205 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":291,"threads_started":1}
I20260929 00:55:32.437702 10493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:32.517467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10371
I20260929 00:55:32.533239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10501
I20260929 00:55:32.537269 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:32.534167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.97.3:44849)
0929 00:55:32.534787 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:32.534794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:32.534824 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:55:32.534884 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:32.534891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:32.535121 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:32.535294 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:32.536455 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:32.536462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:32.536473 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:55:32.536488 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:32.536589 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:55:32.536605 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:32.536784 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:32.536858 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 00:55:32.536957 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":319,"threads_started":1}
I20260929 00:55:32.538609 10623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:32.621402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10501
I20260929 00:55:32.636785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10150
I20260929 00:55:32.637871 10211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:32.736063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10150
2026-09-29T00:55:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2294 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:32 dist-test-slave-pzsx krb5kdc[10646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:32 dist-test-slave-pzsx krb5kdc[10646](info): set up 2 sockets
Sep 29 00:55:32 dist-test-slave-pzsx krb5kdc[10646](info): commencing operation
krb5kdc: starting...
W20260929 00:55:33.821357 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:55:33 dist-test-slave-pzsx krb5kdc[10646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:55:34.046000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:34.365940 10659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:34.366236 10659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:34.366405 10659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:34.373616 10659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:55:34.373751 10659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:34.373853 10659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:34.373965 10659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:55:34.374044 10659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:55:34.383764 10659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:34.385108 10659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:34.386387 10659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:34.398247 10664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:34.398316 10665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:34.398339 10667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:34.398864 10659 server_base.cc:1034] running on GCE node
I20260929 00:55:34.399660 10659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:34.401129 10659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:34.402359 10659 hybrid_clock.cc:648] HybridClock initialized: now 1790643334402332 us; error 46 us; skew 500 ppm
I20260929 00:55:34.402697 10659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:34 dist-test-slave-pzsx krb5kdc[10646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:34.405753 10659 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:55:34.407253 10659 webserver.cc:460] Webserver started at http://127.0.97.62:41559/ using document root <none> and password file <none>
I20260929 00:55:34.407931 10659 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:34.408074 10659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:34.408376 10659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:34.411362 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "b154bfca59ee4a1aa9dee2a6525b9373"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "6a3f0855bc9b3fb7ffbb96632f1e108d"
server_key_iv: "65c69b9b03a7673b45d714f1c0fbba1f"
server_key_version: "encryptionkey@0"
I20260929 00:55:34.412106 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "b154bfca59ee4a1aa9dee2a6525b9373"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "6a3f0855bc9b3fb7ffbb96632f1e108d"
server_key_iv: "65c69b9b03a7673b45d714f1c0fbba1f"
server_key_version: "encryptionkey@0"
I20260929 00:55:34.417205 10659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 00:55:34.420444 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:34.421864 10659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:55:34.422065 10659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "b154bfca59ee4a1aa9dee2a6525b9373"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "6a3f0855bc9b3fb7ffbb96632f1e108d"
server_key_iv: "65c69b9b03a7673b45d714f1c0fbba1f"
server_key_version: "encryptionkey@0"
I20260929 00:55:34.422272 10659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:34.441154 10659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:34.445155 10659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:34.445457 10659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:34.463934 10659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33285
I20260929 00:55:34.463969 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33285 every 8 connection(s)
I20260929 00:55:34.465384 10659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:55:34.467159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10659
I20260929 00:55:34.467392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:55:34.468226 388 external_mini_cluster.cc:1428] Setting key 4015227f96b1159dd591bc4905343aa7
I20260929 00:55:34.470129 10726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 00:55:34 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:34.483508 10726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: Bootstrap starting.
I20260929 00:55:34.486950 10726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:34.488132 10726 log.cc:824] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:34.491179 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:34.470309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.97.62:33285)
0929 00:55:34.471085 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:34.471097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:34.471139 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:55:34.472714 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:34.472736 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:34.473127 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:34.473470 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:34.475170 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:34.476568 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:34.478276 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:34.478904 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:34.479027 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:34.483538 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:34.483546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:34.483561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:34.483599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:34.486631 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:34.487373 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:34.487385 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:34.487392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:34.487547 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:34.488006 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:34.488011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:34.488015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:34.488748 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:34.489122 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:34.489516 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 00:55:34.489927 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":364,"threads_started":1}
I20260929 00:55:34.491698 10726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: No bootstrap required, opened a new log
I20260929 00:55:34.498801 10726 raft_consensus.cc:348] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260929 00:55:34.499153 10726 raft_consensus.cc:374] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:34.499270 10726 raft_consensus.cc:729] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b154bfca59ee4a1aa9dee2a6525b9373, State: Initialized, Role: FOLLOWER
I20260929 00:55:34.499816 10726 consensus_queue.cc:260] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260929 00:55:34.500041 10726 raft_consensus.cc:388] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:55:34.500168 10726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:55:34.500360 10726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:34.502640 10726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260929 00:55:34.503118 10726 leader_election.cc:302] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b154bfca59ee4a1aa9dee2a6525b9373; no voters:
I20260929 00:55:34.503707 10726 leader_election.cc:288] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:55:34.503841 10731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:34.504578 10731 raft_consensus.cc:686] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [term 1 LEADER]: Becoming Leader. State: Replica: b154bfca59ee4a1aa9dee2a6525b9373, State: Running, Role: LEADER
I20260929 00:55:34.505059 10731 consensus_queue.cc:237] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260929 00:55:34.505865 10726 sys_catalog.cc:564] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:55:34.508093 10733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b154bfca59ee4a1aa9dee2a6525b9373. Latest consensus state: current_term: 1 leader_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260929 00:55:34.508523 10733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:34.508301 10732 sys_catalog.cc:455] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b154bfca59ee4a1aa9dee2a6525b9373" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260929 00:55:34.508637 10732 sys_catalog.cc:458] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373 [sys.catalog]: This master's current role is: LEADER
I20260929 00:55:34.510016 10740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:55:34.514628 10740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:55:34.523159 10740 catalog_manager.cc:1347] Generated new cluster ID: 974e142105df4c0780f2e04c6a21258a
I20260929 00:55:34.523278 10740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:55:34.539805 10740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:55:34.541666 10740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:55:34.553354 10740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: Generated new TSK 0
I20260929 00:55:34.554428 10740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:55:34.612326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:34.939761 10752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:34.940023 10752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:34.940182 10752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:34.947283 10752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:34.947389 10752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:34.947602 10752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:55:34.957543 10752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:34.958802 10752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:34.960103 10752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:34.972992 10757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:34.973013 10758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:34.973078 10760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:34.973908 10752 server_base.cc:1034] running on GCE node
I20260929 00:55:34.974538 10752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:34.975485 10752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:34.976709 10752 hybrid_clock.cc:648] HybridClock initialized: now 1790643334976662 us; error 41 us; skew 500 ppm
I20260929 00:55:34.977008 10752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:34 dist-test-slave-pzsx krb5kdc[10646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:34.986616 10752 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:55:34.988050 10752 webserver.cc:460] Webserver started at http://127.0.97.1:39685/ using document root <none> and password file <none>
I20260929 00:55:34.988811 10752 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:34.988910 10752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:34.989163 10752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:34.992446 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "19aacc18d5404c7a99cc3c6f82382e49"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "5bb818ca67b6921c03797c8a5bd31f67"
server_key_iv: "410463a2bf564bdd22ac2d66f8480086"
server_key_version: "encryptionkey@0"
I20260929 00:55:34.993220 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "19aacc18d5404c7a99cc3c6f82382e49"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "5bb818ca67b6921c03797c8a5bd31f67"
server_key_iv: "410463a2bf564bdd22ac2d66f8480086"
server_key_version: "encryptionkey@0"
I20260929 00:55:34.998268 10752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:55:35.001818 10767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.003139 10752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:55:35.003336 10752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "19aacc18d5404c7a99cc3c6f82382e49"
format_stamp: "Formatted at 2026-09-29 00:55:34 on dist-test-slave-pzsx"
server_key: "5bb818ca67b6921c03797c8a5bd31f67"
server_key_iv: "410463a2bf564bdd22ac2d66f8480086"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.003535 10752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:35.031080 10752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:35.034940 10752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:35.035230 10752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:35.036158 10752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:35.037539 10752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:35.037643 10752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.037797 10752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:35.037870 10752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.070676 10752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41333
I20260929 00:55:35.070720 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41333 every 8 connection(s)
I20260929 00:55:35.071942 10752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:55:35.075894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10752
I20260929 00:55:35.076133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:55:35.077005 388 external_mini_cluster.cc:1428] Setting key 719232e04d9cb836295356a071f9354d
Sep 29 00:55:35 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:35.095419 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.074461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57253 (local address 127.0.97.62:33285)
0929 00:55:35.074899 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:35.074905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:35.076097 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 00:55:35.078551 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:35.078556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:35.078639 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:35.078789 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:35.080905 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.081591 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.084016 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.084399 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.084537 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.088637 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:35.088646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:35.088654 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:35.088705 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:35.092177 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.092953 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.092961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.092967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.093107 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.093508 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.093515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.093521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.093897 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:35.094038 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:35.094707 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 00:55:35.094958 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260929 00:55:35.096020 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.075044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.1:57253)
0929 00:55:35.075896 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:35.075954 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:35.078280 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:35.079071 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:35.079082 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:35.079843 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:35.080730 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:35.080759 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.081764 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.081772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:35.083861 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:35.083879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.084650 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.084659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:35.084997 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.085735 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:35.085767 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:35.088398 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:35.092333 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:35.092339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:35.092353 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:35.092826 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:35.093212 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:35.093216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:35.093218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:35.093391 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:35.094009 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:35.094014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:35.094319 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 00:55:35.094539 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 00:55:35.094906 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":319,"threads_started":1}
I20260929 00:55:35.097654 10880 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260929 00:55:35.097995 10880 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:35.098912 10880 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260929 00:55:35.101225 10691 ts_manager.cc:194] Registered new tserver with Master: 19aacc18d5404c7a99cc3c6f82382e49 (127.0.97.1:41333)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:55:35.102710 10691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:55:35.131501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:35.452757 10886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:35.453038 10886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:35.453130 10886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:35.460316 10886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:35.460433 10886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:35.460580 10886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:55:35.470309 10886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:35.471612 10886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:35.472994 10886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:35.486037 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.486205 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.486212 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:35.487258 10886 server_base.cc:1034] running on GCE node
I20260929 00:55:35.487900 10886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:35.488840 10886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:35.490036 10886 hybrid_clock.cc:648] HybridClock initialized: now 1790643335489971 us; error 59 us; skew 500 ppm
I20260929 00:55:35.490420 10886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:35 dist-test-slave-pzsx krb5kdc[10646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:35.493633 10886 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:55:35.495083 10886 webserver.cc:460] Webserver started at http://127.0.97.2:35049/ using document root <none> and password file <none>
I20260929 00:55:35.495743 10886 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:35.495837 10886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:35.496080 10886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:35.498975 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd5f115286034a019e110c5afbaeb4c2"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "cc1980371850bae0e6a31eea16a829bd"
server_key_iv: "23e6ac036fa27f4399103580c881fe95"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.499653 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5f115286034a019e110c5afbaeb4c2"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "cc1980371850bae0e6a31eea16a829bd"
server_key_iv: "23e6ac036fa27f4399103580c881fe95"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.504761 10886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:55:35.508359 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.509883 10886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:55:35.510048 10886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "bd5f115286034a019e110c5afbaeb4c2"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "cc1980371850bae0e6a31eea16a829bd"
server_key_iv: "23e6ac036fa27f4399103580c881fe95"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.510257 10886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:35.540946 10886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:35.544917 10886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:35.545217 10886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:35.546191 10886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:35.547552 10886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:35.547623 10886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.547783 10886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:35.547842 10886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:35.580394 10886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33349
I20260929 00:55:35.580449 11013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33349 every 8 connection(s)
I20260929 00:55:35.581748 10886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:55:35.583074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 10886
I20260929 00:55:35.583292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:55:35.583719 388 external_mini_cluster.cc:1428] Setting key e633aa1d327a90cacc8934c03c820397
Sep 29 00:55:35 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:35.603286 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.584358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42801 (local address 127.0.97.62:33285)
0929 00:55:35.584603 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:35.584609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:35.586148 (+ 1539us) server_negotiation.cc:407] Connection header received
0929 00:55:35.587946 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:35.587954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:35.588039 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:35.588149 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:35.589675 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.590815 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.592546 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.592952 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.593054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.596765 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:35.596773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:35.596781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:35.596822 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:35.600235 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.601005 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.601013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.601020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.601166 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:35.601580 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:35.601588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:35.601595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:35.601953 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:35.602099 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:35.602731 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 00:55:35.602948 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:55:35.604308 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:35.584998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.2:42801)
0929 00:55:35.585979 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:35.586025 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:35.587684 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:35.588351 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:35.588358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:35.588931 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:35.589502 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:35.589519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.590960 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.590964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:35.592359 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:35.592375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:35.593194 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:35.593203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:35.593475 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:35.594197 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:35.594219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:35.596531 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:35.600385 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:35.600392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:35.600404 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:35.600876 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:35.601280 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:35.601283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:35.601285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:35.601463 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:35.602068 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:35.602074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:35.602375 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 00:55:35.602630 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 00:55:35.603161 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":539,"threads_started":1}
I20260929 00:55:35.605836 11014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260929 00:55:35.606194 11014 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:55:35.606968 11014 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260929 00:55:35.608548 10691 ts_manager.cc:194] Registered new tserver with Master: bd5f115286034a019e110c5afbaeb4c2 (127.0.97.2:33349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:55:35.609663 10691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:55:35.635902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:55:35.953590 11019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:55:35.953866 11019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:55:35.953953 11019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:55:35.960978 11019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:55:35.961084 11019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:55:35.961193 11019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:55:35.971050 11019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:55:35.972231 11019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:55:35.973519 11019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:55:35.986236 11027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.986214 11025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:55:35.986632 11024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:55:35.987124 11019 server_base.cc:1034] running on GCE node
I20260929 00:55:35.987672 11019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:55:35.988548 11019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:55:35.989746 11019 hybrid_clock.cc:648] HybridClock initialized: now 1790643335989708 us; error 39 us; skew 500 ppm
I20260929 00:55:35.990048 11019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:55:35 dist-test-slave-pzsx krb5kdc[10646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:55:35.992928 11019 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:55:35.994263 11019 webserver.cc:460] Webserver started at http://127.0.97.3:35421/ using document root <none> and password file <none>
I20260929 00:55:35.994966 11019 fs_manager.cc:359] Metadata directory not provided
I20260929 00:55:35.995103 11019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:55:35.995388 11019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:55:35.998386 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "eba43abaeb134b4aa8bfbe15925e0f7c"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "be26e6b78bd30f19d8b7aac5e46fed6b"
server_key_iv: "ef5dcb3de66bd302964d701f968f017f"
server_key_version: "encryptionkey@0"
I20260929 00:55:35.999115 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "eba43abaeb134b4aa8bfbe15925e0f7c"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "be26e6b78bd30f19d8b7aac5e46fed6b"
server_key_iv: "ef5dcb3de66bd302964d701f968f017f"
server_key_version: "encryptionkey@0"
I20260929 00:55:36.004043 11019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 00:55:36.007681 11034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:36.009121 11019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:55:36.009281 11019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "eba43abaeb134b4aa8bfbe15925e0f7c"
format_stamp: "Formatted at 2026-09-29 00:55:35 on dist-test-slave-pzsx"
server_key: "be26e6b78bd30f19d8b7aac5e46fed6b"
server_key_iv: "ef5dcb3de66bd302964d701f968f017f"
server_key_version: "encryptionkey@0"
I20260929 00:55:36.009466 11019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:55:36.027470 11019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:55:36.031445 11019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:55:36.031741 11019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:55:36.032838 11019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:55:36.037061 11019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:55:36.037151 11019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:36.037281 11019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:55:36.037340 11019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:55:36.070802 11019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34135
I20260929 00:55:36.070834 11146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34135 every 8 connection(s)
I20260929 00:55:36.072003 11019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:55:36.077170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 11019
I20260929 00:55:36.077358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:55:36.077785 388 external_mini_cluster.cc:1428] Setting key 940ccc9da1f92533f29d80efce45c741
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:55:36.091629 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.074658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59277 (local address 127.0.97.62:33285)
0929 00:55:36.074866 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.074873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.076094 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 00:55:36.078062 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.078068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.078160 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.078283 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.079961 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.081057 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.082601 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.082966 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.083069 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.086544 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.086549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.086553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:36.086581 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.088965 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.089620 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.089624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.089626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.089734 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.090114 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.090117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.090119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.090355 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.090460 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.091095 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.091294 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:55:36.092818 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.075165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.3:59277)
0929 00:55:36.075885 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.075921 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.077757 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.078573 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.078582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.079169 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:55:36.079818 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.079835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.081212 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.081216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.082463 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.082475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.083143 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.083147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.083337 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.084009 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:55:36.084034 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:55:36.086339 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:55:36.089099 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:36.089105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:36.089117 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:36.089527 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:36.089849 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:55:36.089852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:55:36.089855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:55:36.090025 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:55:36.090459 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:55:36.090464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:55:36.090751 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.090977 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.091451 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":308,"threads_started":1}
I20260929 00:55:36.094086 11147 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260929 00:55:36.094416 11147 heartbeater.cc:464] Registering TS with master...
I20260929 00:55:36.095157 11147 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260929 00:55:36.096436 10691 ts_manager.cc:194] Registered new tserver with Master: eba43abaeb134b4aa8bfbe15925e0f7c (127.0.97.3:34135)
I20260929 00:55:36.097226 10691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59277
I20260929 00:55:36.105458 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:55:36.105787 10880 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260929 00:55:36.122215 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.110399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.97.62:33285)
0929 00:55:36.110593 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.110601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.111071 (+ 470us) server_negotiation.cc:407] Connection header received
0929 00:55:36.111207 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.111211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.111265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.111336 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.112772 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.113476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.114838 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.115264 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.115354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.117493 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.117498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.117501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:36.117528 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.119943 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.120568 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.120572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.120574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.120683 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.120996 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.120999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.121001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.121274 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.121442 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.121706 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.121860 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:55:36.126104 10691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:55:36.132786 10691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:55:36.155419 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.146430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.0.97.3:34135)
0929 00:55:36.148348 (+ 1918us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.148357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.148387 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:55:36.148477 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.148485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.148703 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.148884 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:36.149493 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.150988 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.153093 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154167 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.154325 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.154568 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.154881 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.155056 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1576,"threads_started":1}
I20260929 00:55:36.155530 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.144464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33349 (local address 127.0.0.1:35782)
0929 00:55:36.146669 (+ 2205us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.146689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.146898 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.147673 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.147680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.147708 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:36.148056 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.148067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.150211 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.150219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.152481 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.152518 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154718 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.154723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.154916 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.154945 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.155025 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.155136 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"mutex_wait_us":42,"thread_start_us":2612,"threads_started":1}
I20260929 00:55:36.156005 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.145836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34135 (local address 127.0.0.1:35960)
0929 00:55:36.147346 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.147366 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.147486 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.148886 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.148892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.148920 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:36.149272 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.149285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.151129 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.151134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.152921 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.152932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154389 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.154395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.154603 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.154666 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.154815 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.155024 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":57,"thread_start_us":908,"threads_started":1}
I20260929 00:55:36.156109 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.145105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.0.97.2:33349)
0929 00:55:36.146252 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.146262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.146780 (+ 518us) server_negotiation.cc:407] Connection header received
0929 00:55:36.147112 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.147123 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.147417 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.147592 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:36.148288 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.150073 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.152707 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154191 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.154417 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.154602 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.155550 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.155741 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":875,"threads_started":1}
I20260929 00:55:36.157426 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.144465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.97.1:41333)
0929 00:55:36.147691 (+ 3226us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.147699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.147724 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:55:36.147832 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.147840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.148064 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.148840 (+ 776us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:36.149089 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.150731 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.153307 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154734 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.156628 (+ 1894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.156850 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.156938 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.157098 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3062,"thread_start_us":2877,"threads_started":1}
I20260929 00:55:36.156196 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.144184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41333 (local address 127.0.0.1:43728)
0929 00:55:36.145251 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.145295 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.145496 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.148260 (+ 2764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.148267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.148300 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:36.148649 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.148658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.150875 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.150880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.153126 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.153143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.154988 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.154996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.155147 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.155189 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.155263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.155398 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":4,"thread_start_us":655,"threads_started":1}
I20260929 00:55:36.159710 11082 tablet_service.cc:1467] Processing CreateTablet for tablet d6eb203338824523a54f78bdf4b28593 (DEFAULT_TABLE table=test-table [id=8fc2c5dbe14b4e37b4284107c826c095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:36.159770 10949 tablet_service.cc:1467] Processing CreateTablet for tablet d6eb203338824523a54f78bdf4b28593 (DEFAULT_TABLE table=test-table [id=8fc2c5dbe14b4e37b4284107c826c095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:36.161054 11082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eb203338824523a54f78bdf4b28593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:36.161106 10949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eb203338824523a54f78bdf4b28593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:36.161414 10815 tablet_service.cc:1467] Processing CreateTablet for tablet d6eb203338824523a54f78bdf4b28593 (DEFAULT_TABLE table=test-table [id=8fc2c5dbe14b4e37b4284107c826c095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:55:36.162725 10815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eb203338824523a54f78bdf4b28593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:55:36.172643 11165 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Bootstrap starting.
I20260929 00:55:36.176004 11165 tablet_bootstrap.cc:654] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:36.177062 11166 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Bootstrap starting.
I20260929 00:55:36.177562 11165 log.cc:824] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:36.179045 11167 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Bootstrap starting.
I20260929 00:55:36.180912 11165 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: No bootstrap required, opened a new log
I20260929 00:55:36.180915 11166 tablet_bootstrap.cc:654] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:36.181193 11165 ts_tablet_manager.cc:1397] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 00:55:36.182134 11166 log.cc:824] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:36.182476 11167 tablet_bootstrap.cc:654] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Neither blocks nor log segments found. Creating new log.
I20260929 00:55:36.183590 11167 log.cc:824] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Log is configured to *not* fsync() on all Append() calls
I20260929 00:55:36.185454 11166 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: No bootstrap required, opened a new log
I20260929 00:55:36.185779 11166 ts_tablet_manager.cc:1397] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 00:55:36.189219 11165 raft_consensus.cc:348] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.189658 11165 raft_consensus.cc:374] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:36.189790 11165 raft_consensus.cc:729] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba43abaeb134b4aa8bfbe15925e0f7c, State: Initialized, Role: FOLLOWER
I20260929 00:55:36.190399 11165 consensus_queue.cc:260] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.191183 11167 tablet_bootstrap.cc:492] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: No bootstrap required, opened a new log
I20260929 00:55:36.191519 11147 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260929 00:55:36.191960 11165 ts_tablet_manager.cc:1428] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 00:55:36.192317 11167 ts_tablet_manager.cc:1397] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260929 00:55:36.195567 11166 raft_consensus.cc:348] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.196022 11166 raft_consensus.cc:374] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:36.196156 11166 raft_consensus.cc:729] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5f115286034a019e110c5afbaeb4c2, State: Initialized, Role: FOLLOWER
I20260929 00:55:36.196813 11166 consensus_queue.cc:260] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.198850 11014 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260929 00:55:36.199360 11166 ts_tablet_manager.cc:1428] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260929 00:55:36.202030 11167 raft_consensus.cc:348] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.202389 11167 raft_consensus.cc:374] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:55:36.202481 11167 raft_consensus.cc:729] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aacc18d5404c7a99cc3c6f82382e49, State: Initialized, Role: FOLLOWER
I20260929 00:55:36.203065 11167 consensus_queue.cc:260] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.204373 11167 ts_tablet_manager.cc:1428] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 00:55:36.223767 11173 raft_consensus.cc:482] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:55:36.224078 11173 raft_consensus.cc:504] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.225816 11173 leader_election.cc:288] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd5f115286034a019e110c5afbaeb4c2 (127.0.97.2:33349), eba43abaeb134b4aa8bfbe15925e0f7c (127.0.97.3:34135)
I20260929 00:55:36.232263 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.226362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39199 (local address 127.0.97.3:34135)
0929 00:55:36.226595 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.226604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.227149 (+ 545us) server_negotiation.cc:407] Connection header received
0929 00:55:36.227306 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.227312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.227388 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.227472 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:36.227996 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.228818 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.230653 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.231367 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.231509 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.231607 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.231876 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.231992 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:55:36.232326 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.226219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46581 (local address 127.0.97.2:33349)
0929 00:55:36.226440 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.226448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.227317 (+ 869us) server_negotiation.cc:407] Connection header received
0929 00:55:36.227462 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.227465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.227521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.227610 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:55:36.228211 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.228948 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.230679 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.231425 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.231574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.231670 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.231934 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.232052 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:55:36.232427 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.226213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33349 (local address 127.0.97.1:46581)
0929 00:55:36.227187 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.227208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.227340 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.227653 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.227659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.227685 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:36.228065 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.228073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.229104 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.229109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.230534 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.230545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.231649 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.231655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.231784 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.231811 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.231911 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.232024 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":24,"thread_start_us":656,"threads_started":1}
I20260929 00:55:36.232949 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.226181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34135 (local address 127.0.97.1:39199)
0929 00:55:36.227014 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 00:55:36.227043 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 00:55:36.227173 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:55:36.227481 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:55:36.227485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:55:36.227505 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:55:36.227858 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.227867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.228941 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.228949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.230498 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:55:36.230512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.231566 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.231571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:55:36.231730 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.231752 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:55:36.231852 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:55:36.231969 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":32,"thread_start_us":432,"threads_started":1}
I20260929 00:55:36.233414 10969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eb203338824523a54f78bdf4b28593" candidate_uuid: "19aacc18d5404c7a99cc3c6f82382e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5f115286034a019e110c5afbaeb4c2" is_pre_election: true
I20260929 00:55:36.234000 10969 raft_consensus.cc:2454] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aacc18d5404c7a99cc3c6f82382e49 in term 0.
I20260929 00:55:36.233942 11102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eb203338824523a54f78bdf4b28593" candidate_uuid: "19aacc18d5404c7a99cc3c6f82382e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" is_pre_election: true
I20260929 00:55:36.234440 11102 raft_consensus.cc:2454] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aacc18d5404c7a99cc3c6f82382e49 in term 0.
I20260929 00:55:36.234738 10771 leader_election.cc:302] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aacc18d5404c7a99cc3c6f82382e49, bd5f115286034a019e110c5afbaeb4c2; no voters:
I20260929 00:55:36.235162 11173 raft_consensus.cc:2781] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:55:36.235318 11173 raft_consensus.cc:482] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:55:36.235428 11173 raft_consensus.cc:3037] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:36.237625 11173 raft_consensus.cc:504] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.238193 11173 leader_election.cc:288] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [CANDIDATE]: Term 1 election: Requested vote from peers bd5f115286034a019e110c5afbaeb4c2 (127.0.97.2:33349), eba43abaeb134b4aa8bfbe15925e0f7c (127.0.97.3:34135)
I20260929 00:55:36.238549 10969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eb203338824523a54f78bdf4b28593" candidate_uuid: "19aacc18d5404c7a99cc3c6f82382e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5f115286034a019e110c5afbaeb4c2"
I20260929 00:55:36.238639 11102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eb203338824523a54f78bdf4b28593" candidate_uuid: "19aacc18d5404c7a99cc3c6f82382e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c"
I20260929 00:55:36.238740 10969 raft_consensus.cc:3037] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:36.238855 11102 raft_consensus.cc:3037] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:55:36.241047 10969 raft_consensus.cc:2454] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aacc18d5404c7a99cc3c6f82382e49 in term 1.
I20260929 00:55:36.241065 11102 raft_consensus.cc:2454] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aacc18d5404c7a99cc3c6f82382e49 in term 1.
I20260929 00:55:36.241425 10771 leader_election.cc:302] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19aacc18d5404c7a99cc3c6f82382e49, bd5f115286034a019e110c5afbaeb4c2, eba43abaeb134b4aa8bfbe15925e0f7c; no voters:
I20260929 00:55:36.241709 11173 raft_consensus.cc:2781] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:55:36.242168 11173 raft_consensus.cc:686] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 1 LEADER]: Becoming Leader. State: Replica: 19aacc18d5404c7a99cc3c6f82382e49, State: Running, Role: LEADER
I20260929 00:55:36.242622 11173 consensus_queue.cc:237] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } }
I20260929 00:55:36.247134 10691 catalog_manager.cc:5495] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19aacc18d5404c7a99cc3c6f82382e49 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "19aacc18d5404c7a99cc3c6f82382e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aacc18d5404c7a99cc3c6f82382e49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20260929 00:55:36.279171 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.274336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.0.97.1:41333)
0929 00:55:36.274517 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.274522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.274590 (+ 68us) server_negotiation.cc:407] Connection header received
0929 00:55:36.274771 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.274775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.274849 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.274974 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:55:36.275297 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.276060 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.277374 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.277765 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.277851 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.278263 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:55:36.278754 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:55:36.278867 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.278922 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.278982 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:55:36.287271 11102 raft_consensus.cc:1261] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 1 FOLLOWER]: Refusing update from remote peer 19aacc18d5404c7a99cc3c6f82382e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:36.287264 10969 raft_consensus.cc:1261] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 1 FOLLOWER]: Refusing update from remote peer 19aacc18d5404c7a99cc3c6f82382e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:55:36.288184 11176 consensus_queue.cc:1035] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5f115286034a019e110c5afbaeb4c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:36.288574 11173 consensus_queue.cc:1035] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba43abaeb134b4aa8bfbe15925e0f7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:55:36.299484 11179 mvcc.cc:205] Tried to move back new op lower bound from 7334475105422708736 to 7334475105253462016. Current Snapshot: MvccSnapshot[applied={T|T < 7334475105422708736}]
I20260929 00:55:36.317878 10690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:36.318161 10690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:55:36.321723 10690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b154bfca59ee4a1aa9dee2a6525b9373: Sending DeleteTablet for 3 replicas of tablet d6eb203338824523a54f78bdf4b28593
I20260929 00:55:36.322850 10815 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eb203338824523a54f78bdf4b28593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43728
I20260929 00:55:36.322829 10949 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eb203338824523a54f78bdf4b28593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35782
I20260929 00:55:36.323060 11082 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eb203338824523a54f78bdf4b28593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:55:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35960
I20260929 00:55:36.324884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10752
I20260929 00:55:36.333855 11192 tablet_replica.cc:330] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: stopping tablet replica
I20260929 00:55:36.334287 11192 raft_consensus.cc:2229] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:36.334868 11192 raft_consensus.cc:2258] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:36.337045 11192 ts_tablet_manager.cc:1905] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:36.340647 11190 tablet_replica.cc:330] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: stopping tablet replica
I20260929 00:55:36.341132 11190 raft_consensus.cc:2229] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:55:36.341661 11190 raft_consensus.cc:2258] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:36.343865 11190 ts_tablet_manager.cc:1905] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:55:36.345662 11192 ts_tablet_manager.cc:1918] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:36.345913 11192 log.cc:1196] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/d6eb203338824523a54f78bdf4b28593
I20260929 00:55:36.346529 11192 ts_tablet_manager.cc:1939] T d6eb203338824523a54f78bdf4b28593 P eba43abaeb134b4aa8bfbe15925e0f7c: Deleting consensus metadata
I20260929 00:55:36.348615 10677 catalog_manager.cc:4841] TS eba43abaeb134b4aa8bfbe15925e0f7c (127.0.97.3:34135): tablet d6eb203338824523a54f78bdf4b28593 (table test-table [id=8fc2c5dbe14b4e37b4284107c826c095]) successfully deleted
I20260929 00:55:36.351744 11191 tablet_replica.cc:330] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: stopping tablet replica
I20260929 00:55:36.351958 11190 ts_tablet_manager.cc:1918] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:36.352128 11190 log.cc:1196] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/d6eb203338824523a54f78bdf4b28593
I20260929 00:55:36.352524 11191 raft_consensus.cc:2229] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:55:36.352705 11190 ts_tablet_manager.cc:1939] T d6eb203338824523a54f78bdf4b28593 P bd5f115286034a019e110c5afbaeb4c2: Deleting consensus metadata
I20260929 00:55:36.353382 11191 raft_consensus.cc:2258] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:55:36.354427 10678 catalog_manager.cc:4841] TS bd5f115286034a019e110c5afbaeb4c2 (127.0.97.2:33349): tablet d6eb203338824523a54f78bdf4b28593 (table test-table [id=8fc2c5dbe14b4e37b4284107c826c095]) successfully deleted
I20260929 00:55:36.355525 11191 ts_tablet_manager.cc:1905] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:55:36.363134 11191 ts_tablet_manager.cc:1918] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:55:36.363317 11191 log.cc:1196] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/d6eb203338824523a54f78bdf4b28593
I20260929 00:55:36.363924 11191 ts_tablet_manager.cc:1939] T d6eb203338824523a54f78bdf4b28593 P 19aacc18d5404c7a99cc3c6f82382e49: Deleting consensus metadata
I20260929 00:55:36.365830 10677 catalog_manager.cc:4841] TS 19aacc18d5404c7a99cc3c6f82382e49 (127.0.97.1:41333): tablet d6eb203338824523a54f78bdf4b28593 (table test-table [id=8fc2c5dbe14b4e37b4284107c826c095]) successfully deleted
I20260929 00:55:36.366425 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.326378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.97.1:41333)
0929 00:55:36.326572 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.326580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.350956 (+ 24376us) server_negotiation.cc:407] Connection header received
0929 00:55:36.351161 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.351167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.351256 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.351332 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.353138 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.354473 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.355816 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.356298 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.356438 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.360165 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.360175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.360192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:36.360227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.363508 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.364218 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.364230 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.364232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.364364 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.364795 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.364800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.364808 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.365139 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.365439 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.365722 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.365950 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":40}
I20260929 00:55:36.367103 10875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:36.465638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10752
I20260929 00:55:36.485021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10886
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:55:36.498344 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.485768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.0.97.2:33349)
0929 00:55:36.485940 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.485944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.485998 (+ 54us) server_negotiation.cc:407] Connection header received
0929 00:55:36.486138 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.486142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.486205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.486288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.487426 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.488125 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.489333 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.489691 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.489782 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.493483 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.493498 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.493514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:36.493547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.496017 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.496718 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.496725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.496727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.496846 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.497141 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.497144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.497146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.497501 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.497704 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.497945 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.498090 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:55:36.498946 11009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:36.586229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10886
I20260929 00:55:36.603364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 11019
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[10646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:55:36.617769 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:55:36.604093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.97.3:34135)
0929 00:55:36.604272 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:55:36.604276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:55:36.604296 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:55:36.604387 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:55:36.604390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:55:36.604449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:55:36.604618 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:55:36.605720 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.606513 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.607646 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:55:36.608014 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:55:36.608101 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:55:36.612294 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:55:36.612328 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:55:36.612345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:55:36.612381 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:55:36.615346 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.615933 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.615941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.615943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.616070 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:55:36.616384 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:55:36.616387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:55:36.616389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:55:36.616803 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:55:36.617005 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 00:55:36.617270 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 00:55:36.617423 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:55:36.618422 11142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:36.706646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 11019
I20260929 00:55:36.723281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 10659
I20260929 00:55:36.724414 10721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:55:36.830921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 10659
2026-09-29T00:55:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4096 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[11200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[11200](info): set up 2 sockets
Sep 29 00:55:36 dist-test-slave-pzsx krb5kdc[11200](info): commencing operation
krb5kdc: starting...
W20260929 00:55:37.925370 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:55:38 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:55:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 00:55:38.241566 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiPz2y/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/postgres -l logfile start
2026-09-29 00:55:40.623 UTC [11230] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:55:40.623 UTC [11230] LOG: listening on IPv4 address "127.0.97.20", port 34393
2026-09-29 00:55:40.627 UTC [11230] LOG: listening on Unix socket "/tmp/.s.PGSQL.34393"
2026-09-29 00:55:40.649 UTC [11233] LOG: database system was shut down at 2026-09-29 00:55:39 UTC
2026-09-29 00:55:40.652 UTC [11230] LOG: database system is ready to accept connections
I20260929 00:55:41.617509 388 mini_postgres.cc:96] Postgres bound to 34393
2026-09-29 00:55:41.634 UTC [11241] FATAL: database "slave" does not exist
127.0.97.20:34393 - accepting connections
I20260929 00:55:41.634887 388 mini_ranger.cc:161] Starting Ranger...
I20260929 00:55:41.661343 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:55:42.134946 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:55:42.135071 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin
2026-09-29 00:55:42,425 [I] DB FLAVOR :POSTGRES
2026-09-29 00:55:42,425 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:55:42,425 [I] Checking connection..
2026-09-29 00:55:42,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:55:42,678 [I] Checking connection passed.
2026-09-29 00:55:42,679 [I] --------- Verifying version history table ---------
2026-09-29 00:55:42,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:55:42,937 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:55:42,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:55:43,192 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:55:43,193 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:55:43,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:55:43.418 UTC [11305] WARNING: there is no transaction in progress
2026-09-29 00:55:43,425 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:55:43,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:55:43,684 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:55:43,684 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:55:43,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:55:43,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:55:44,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pzsx.c.gcp-upstream.internal','N') ;"
2026-09-29 00:55:44,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:55:44,680 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:55:44,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:55:44,921 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:55:44,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:55:45,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:55:45,424 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:55:45,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:55:46.363 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.379 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.395 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.409 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.424 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.440 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.490 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.497 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.510 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.520 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.530 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.542 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.551 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.557 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.564 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:46.570 UTC [11413] WARNING: there is no transaction in progress
2026-09-29 00:55:47,167 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:55:47,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:55:47,402 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:55:47,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:55:47,629 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:55:47,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:55:47,859 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:55:47,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:55:48,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:55:48,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pzsx.c.gcp-upstream.internal';"
2026-09-29 00:55:48,592 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:55:48,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:55:48,833 [I] DB_PATCHES have already been applied
I20260929 00:55:48.839076 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiPz2y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:55:48.839190 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 00:55:48.853960 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:53775
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:55:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 00:55:49.994206 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.001s sys 0.038s
Sep 29, 2026 12:56:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:56:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 12:56:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:56:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:56:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:56:10.390818 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 00:56:10 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 00:56:10.587728 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 00:56:10.610487 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 00:56:11.484716 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 00:56:11.484881 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms
I20260929 00:56:11.485016 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34393
2026-09-29 00:56:11,891 [I] DB FLAVOR :POSTGRES
2026-09-29 00:56:11,891 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:56:11,891 [I] Checking connection
2026-09-29 00:56:11,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 00:56:12,124 [I] connection success
2026-09-29 00:56:12,125 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 00:56:12,125 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 00:56:12,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 00:56:12,357 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 00:56:12,357 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:56:12,357 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 00:56:12,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 00:56:12,619 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 00:56:13.064637 388 mini_ranger_kms.cc:310] Created kms service
I20260929 00:56:13.258728 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 00:56:13.303030 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 00:56:13 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:13.674124 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 00:56:13.674480 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 00:56:13.674538 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 00:56:13.695785 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11593
I20260929 00:56:13.695971 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:37415
Sep 29, 2026 12:56:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:56:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:56:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 12:56:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 12:56:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 12:56:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 00:56:15 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:15.941241 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.354s user 0.121s sys 0.308s
I20260929 00:56:15.941497 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 00:56:15.941749 388 mini_ranger_kms.cc:401] 127.0.97.20:37415/kms/v1/keys
Sep 29 00:56:16 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 00:56:16 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643375, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 00:56:18 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:56:18.905176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33721
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33721
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37415/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:56:19.236418 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:19.236780 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:19.236930 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:19.244092 11644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:56:19.244719 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:19.244832 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:19.244935 11644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:56:19.245012 11644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:56:19.255019 11644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:37415/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33721
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:56:19.256341 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:19.257653 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:19.269515 11649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:19.269712 11650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:19.269690 11652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:19.270713 11644 server_base.cc:1034] running on GCE node
I20260929 00:56:19.271477 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:19.272907 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:19.274129 11644 hybrid_clock.cc:648] HybridClock initialized: now 1790643379274099 us; error 43 us; skew 500 ppm
I20260929 00:56:19.274467 11644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:56:19 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:19.284449 11644 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:56:19.285878 11644 webserver.cc:460] Webserver started at http://127.0.97.62:37983/ using document root <none> and password file <none>
I20260929 00:56:19.286505 11644 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:19.286638 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:19.286937 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:56:19 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:19.351967 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "00029d120f1048bc94d3720aa2d8dae5"
format_stamp: "Formatted at 2026-09-29 00:56:19 on dist-test-slave-pzsx"
server_key: "022b1a3d2278819cf449b974cf21a933"
server_key_iv: "10b1ab85b94b75e0382ca79dc68a4bfc"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:19.353055 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "00029d120f1048bc94d3720aa2d8dae5"
format_stamp: "Formatted at 2026-09-29 00:56:19 on dist-test-slave-pzsx"
server_key: "022b1a3d2278819cf449b974cf21a933"
server_key_iv: "10b1ab85b94b75e0382ca79dc68a4bfc"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:19.360078 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 00:56:19.380725 11659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:19.382607 11644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:56:19.382818 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "00029d120f1048bc94d3720aa2d8dae5"
format_stamp: "Formatted at 2026-09-29 00:56:19 on dist-test-slave-pzsx"
server_key: "022b1a3d2278819cf449b974cf21a933"
server_key_iv: "10b1ab85b94b75e0382ca79dc68a4bfc"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:19.383086 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:19.403091 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:19.407887 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:19.408210 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:19.429536 11644 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33721
I20260929 00:56:19.429577 11710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33721 every 8 connection(s)
I20260929 00:56:19.431277 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:56:19.433065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 11644
I20260929 00:56:19.433265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:56:19.434044 388 external_mini_cluster.cc:1428] Setting key 280130170852abb6de63935ee50b8319
I20260929 00:56:19.436208 11711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:19.453691 11711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: Bootstrap starting.
I20260929 00:56:19.457521 11711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:19.458645 11711 log.cc:824] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:19.461896 11711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: No bootstrap required, opened a new log
Sep 29 00:56:19 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:56:19.473721 11711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } }
I20260929 00:56:19.474218 11711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:19.475014 11711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00029d120f1048bc94d3720aa2d8dae5, State: Initialized, Role: FOLLOWER
I20260929 00:56:19.475400 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:19.436084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.97.62:33721)
0929 00:56:19.449249 (+ 13165us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:19.449272 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:19.450331 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 00:56:19.453381 (+ 3050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:19.453410 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:19.453967 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:19.454411 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:19.455951 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:19.458035 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:19.459783 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:19.460448 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:19.460692 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:19.464757 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:19.464765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:19.464778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:19.464815 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:19.468832 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:19.469531 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:19.469538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:19.469545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:19.469681 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:19.470057 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:19.470067 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:19.470071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:19.470601 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:19.470914 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:19.473306 (+ 2392us) server_negotiation.cc:299] Negotiation successful
0929 00:56:19.473753 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12795,"thread_start_us":12512,"threads_started":1}
I20260929 00:56:19.475879 11711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } }
I20260929 00:56:19.476215 11711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:56:19.476387 11711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:56:19.476649 11711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:19.480558 11711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } }
I20260929 00:56:19.481410 11711 leader_election.cc:302] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00029d120f1048bc94d3720aa2d8dae5; no voters:
I20260929 00:56:19.482335 11716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:19.482555 11711 leader_election.cc:288] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:56:19.484760 11711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:56:19.489427 11716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [term 1 LEADER]: Becoming Leader. State: Replica: 00029d120f1048bc94d3720aa2d8dae5, State: Running, Role: LEADER
I20260929 00:56:19.490077 11716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } }
I20260929 00:56:19.493287 11718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00029d120f1048bc94d3720aa2d8dae5. Latest consensus state: current_term: 1 leader_uuid: "00029d120f1048bc94d3720aa2d8dae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } } }
I20260929 00:56:19.493664 11718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:19.501703 11723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:56:19.501642 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00029d120f1048bc94d3720aa2d8dae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00029d120f1048bc94d3720aa2d8dae5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33721 } } }
I20260929 00:56:19.501968 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5 [sys.catalog]: This master's current role is: LEADER
I20260929 00:56:19.511543 11723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:56:19.535557 11723 catalog_manager.cc:1347] Generated new cluster ID: 8419178b4ab54d8fad9e85a17f6a8a49
I20260929 00:56:19.535696 11723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:56:19.545022 11711 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 00:56:19.577509 11723 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:56:19.580016 11723 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:56:19.600980 11723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: Generated new TSK 0
I20260929 00:56:19.602205 11723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:56:20.341168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37415/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:56:20 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 00:56:20.787920 11757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:20.788339 11757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:20.788568 11757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:20.801874 11757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:20.802018 11757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:20.802238 11757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:56:20.820122 11757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:37415/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:56:20.822042 11757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:20.823984 11757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:20.840787 11770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:20.840935 11767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:20.841300 11768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:20.842269 11757 server_base.cc:1034] running on GCE node
I20260929 00:56:20.842912 11757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:20.848824 11757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:20.850095 11757 hybrid_clock.cc:648] HybridClock initialized: now 1790643380850022 us; error 71 us; skew 500 ppm
I20260929 00:56:20.850436 11757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:56:20 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:20.853751 11757 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:56:20.872354 11757 webserver.cc:460] Webserver started at http://127.0.97.1:43529/ using document root <none> and password file <none>
I20260929 00:56:20.873185 11757 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:20.873345 11757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:20.873736 11757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:56:20 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:20.898250 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "ed5c2fc9420f4564954bc59e3670509b"
format_stamp: "Formatted at 2026-09-29 00:56:20 on dist-test-slave-pzsx"
server_key: "d861e57f585752d0846fb6b98989d4d1"
server_key_iv: "ab2737c9930a22250dfdf39b3cd5896e"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:20.899389 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ed5c2fc9420f4564954bc59e3670509b"
format_stamp: "Formatted at 2026-09-29 00:56:20 on dist-test-slave-pzsx"
server_key: "d861e57f585752d0846fb6b98989d4d1"
server_key_iv: "ab2737c9930a22250dfdf39b3cd5896e"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:20.909404 11757 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.006s
I20260929 00:56:20.941275 11777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:20.943671 11757 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.012s
I20260929 00:56:20.943935 11757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "ed5c2fc9420f4564954bc59e3670509b"
format_stamp: "Formatted at 2026-09-29 00:56:20 on dist-test-slave-pzsx"
server_key: "d861e57f585752d0846fb6b98989d4d1"
server_key_iv: "ab2737c9930a22250dfdf39b3cd5896e"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:20.944269 11757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:21.006397 11757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:21.013872 11757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:21.014390 11757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:21.033349 11757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:21.039341 11757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:21.039461 11757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:21.039623 11757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:21.039747 11757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 00:56:21 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:21.469846 11757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44937
I20260929 00:56:21.469858 11889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44937 every 8 connection(s)
I20260929 00:56:21.472059 11757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:56:21.478753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 11757
I20260929 00:56:21.478962 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:56:21.479736 388 external_mini_cluster.cc:1428] Setting key f24bcf55727d78faae459c93a3a3fefb
Sep 29 00:56:21 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:56:21.536963 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:21.476935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52611 (local address 127.0.97.62:33721)
0929 00:56:21.505521 (+ 28586us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:21.505531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:21.505558 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:56:21.505639 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:21.505648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:21.505789 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:21.505925 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:21.519634 (+ 13709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:21.520973 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:21.523126 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:21.523926 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:21.524091 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:21.528877 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:21.528887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:21.528892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:21.528934 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:21.532744 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:21.533806 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:21.533814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:21.533820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:21.533968 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:21.534554 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:21.534562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:21.534568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:21.534961 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:21.535101 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:21.536158 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 00:56:21.536415 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28391,"thread_start_us":29116,"threads_started":1}
I20260929 00:56:21.539685 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:21.476502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33721 (local address 127.0.97.1:52611)
0929 00:56:21.499381 (+ 22879us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:21.499461 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:21.503463 (+ 4002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:21.517585 (+ 14122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:21.517600 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:21.518344 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:21.519334 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:21.519369 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:21.521208 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:21.521215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:21.522960 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:21.522972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:21.524479 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:21.524488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:21.524864 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:21.525749 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:21.525785 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:21.528604 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:21.532926 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:21.532938 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:21.532960 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:21.533647 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:21.534113 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:21.534120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:21.534127 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:21.534391 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:21.535226 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:21.535237 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:21.535715 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 00:56:21.537550 (+ 1835us) client_negotiation.cc:240] Negotiation successful
0929 00:56:21.538142 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21699,"thread_start_us":26699,"threads_started":1}
I20260929 00:56:21.541946 11890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33721
I20260929 00:56:21.542280 11890 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:21.543119 11890 heartbeater.cc:511] Master 127.0.97.62:33721 requested a full tablet report, sending...
I20260929 00:56:21.545471 11676 ts_manager.cc:194] Registered new tserver with Master: ed5c2fc9420f4564954bc59e3670509b (127.0.97.1:44937)
I20260929 00:56:21.547751 11676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:56:21.567725 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37415/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:56:21 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 00:56:21.995296 11896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:21.995579 11896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:21.995740 11896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:22.003113 11896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:22.003225 11896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:22.003473 11896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:56:22.013448 11896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:37415/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:56:22.014720 11896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:22.016026 11896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:22.048298 11910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:22.048314 11911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:22.048906 11913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:22.049414 11896 server_base.cc:1034] running on GCE node
I20260929 00:56:22.050091 11896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:22.051443 11896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:22.052713 11896 hybrid_clock.cc:648] HybridClock initialized: now 1790643382052640 us; error 58 us; skew 500 ppm
I20260929 00:56:22.053043 11896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:22.056265 11896 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:56:22.058315 11896 webserver.cc:460] Webserver started at http://127.0.97.2:37909/ using document root <none> and password file <none>
I20260929 00:56:22.059022 11896 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:22.059197 11896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:22.059544 11896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:22.081768 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "741076093dae49cb956187da73e792ca"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "8d3ac417f4228c0ea4830e97f1a9762b"
server_key_iv: "2d1408c4582180ced5b58ef19dd589bd"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.082892 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "741076093dae49cb956187da73e792ca"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "8d3ac417f4228c0ea4830e97f1a9762b"
server_key_iv: "2d1408c4582180ced5b58ef19dd589bd"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.106624 11896 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.002s sys 0.019s
I20260929 00:56:22.127383 11920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:22.130106 11896 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 00:56:22.130304 11896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "741076093dae49cb956187da73e792ca"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "8d3ac417f4228c0ea4830e97f1a9762b"
server_key_iv: "2d1408c4582180ced5b58ef19dd589bd"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.130580 11896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:22.160018 11896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:22.165443 11896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:22.165798 11896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:22.167109 11896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:22.169226 11896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:22.169363 11896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 00:56:22.169512 11896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:22.169602 11896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:22.219949 11896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39653
I20260929 00:56:22.219970 12032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39653 every 8 connection(s)
I20260929 00:56:22.221706 11896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:56:22.227241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 11896
I20260929 00:56:22.227432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:56:22.228010 388 external_mini_cluster.cc:1428] Setting key a710ee3dde08a6248ea924bddb835c01
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:56:22.266072 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.224802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55427 (local address 127.0.97.62:33721)
0929 00:56:22.239010 (+ 14208us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.239019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.242048 (+ 3029us) server_negotiation.cc:407] Connection header received
0929 00:56:22.244085 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.244094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.244190 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.244297 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:22.246755 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.248132 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.250953 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.251722 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.251835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.257170 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:22.257180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:22.257187 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:22.257236 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:22.260813 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.261717 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.261723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.261726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.261883 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.263534 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.263543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.263549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.263949 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:22.264101 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.265240 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.265530 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14042,"thread_start_us":14213,"threads_started":1}
I20260929 00:56:22.268965 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.225232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33721 (local address 127.0.97.2:55427)
0929 00:56:22.239725 (+ 14493us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:22.239795 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:22.243747 (+ 3952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:22.244943 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:22.244959 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:22.245682 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:22.246550 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.246579 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.248357 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.248366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.250779 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.250797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.252252 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.252276 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.252620 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.253525 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:22.253561 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:22.256892 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:22.261053 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:22.261062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:22.261081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:22.261569 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:22.262021 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:22.262029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:22.262035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:22.263394 (+ 1359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:22.264202 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:22.264212 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:22.264757 (+ 545us) client_negotiation.cc:769] Sending connection context
0929 00:56:22.266400 (+ 1643us) client_negotiation.cc:240] Negotiation successful
0929 00:56:22.267003 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14107,"thread_start_us":13870,"threads_started":1}
I20260929 00:56:22.270761 12033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33721
I20260929 00:56:22.271104 12033 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:22.272015 12033 heartbeater.cc:511] Master 127.0.97.62:33721 requested a full tablet report, sending...
I20260929 00:56:22.273731 11676 ts_manager.cc:194] Registered new tserver with Master: 741076093dae49cb956187da73e792ca (127.0.97.2:39653)
I20260929 00:56:22.274719 11676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:56:22.293642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37415/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:56:22.552402 11890 heartbeater.cc:503] Master 127.0.97.62:33721 was elected leader, sending a full tablet report...
W20260929 00:56:22.630055 12039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:56:22.630344 12039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:56:22.630508 12039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:56:22.638242 12039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:56:22.638392 12039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:56:22.638938 12039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:56:22.648366 12039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:37415/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:56:22.649660 12039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:56:22.651012 12039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:56:22.664350 12045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:22.664479 12044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:56:22.664464 12047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:56:22.665318 12039 server_base.cc:1034] running on GCE node
I20260929 00:56:22.665859 12039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:56:22.666754 12039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:56:22.667982 12039 hybrid_clock.cc:648] HybridClock initialized: now 1790643382667945 us; error 52 us; skew 500 ppm
I20260929 00:56:22.668280 12039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:56:22.671442 12039 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:56:22.672859 12039 webserver.cc:460] Webserver started at http://127.0.97.3:44441/ using document root <none> and password file <none>
I20260929 00:56:22.673524 12039 fs_manager.cc:359] Metadata directory not provided
I20260929 00:56:22.673667 12039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:56:22.673983 12039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:56:22.687202 12039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "902f2a99ae3b4753a0903de627c6109d"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "7a8fd4cae5f8806e779645ba61423ec0"
server_key_iv: "b394778aa3a7b41402622c86bf13b861"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.688099 12039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "902f2a99ae3b4753a0903de627c6109d"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "7a8fd4cae5f8806e779645ba61423ec0"
server_key_iv: "b394778aa3a7b41402622c86bf13b861"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.693686 12039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 00:56:22.707881 12054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:22.709360 12039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:56:22.709560 12039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "902f2a99ae3b4753a0903de627c6109d"
format_stamp: "Formatted at 2026-09-29 00:56:22 on dist-test-slave-pzsx"
server_key: "7a8fd4cae5f8806e779645ba61423ec0"
server_key_iv: "b394778aa3a7b41402622c86bf13b861"
server_key_version: "kuduclusterkey@0"
I20260929 00:56:22.709796 12039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:56:22.736760 12039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:56:22.740465 12039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:56:22.740783 12039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:56:22.742200 12039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:56:22.743564 12039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:56:22.743670 12039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:22.743817 12039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:56:22.743899 12039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:56:22.778455 12039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46339
I20260929 00:56:22.778476 12166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46339 every 8 connection(s)
I20260929 00:56:22.779778 12039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:56:22.786335 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 12039
I20260929 00:56:22.786520 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:56:22.786962 388 external_mini_cluster.cc:1428] Setting key 50a5fee0cfd2aa445dbc6f904b6814ea
Sep 29 00:56:22 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:56:22.799242 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.782156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56769 (local address 127.0.97.62:33721)
0929 00:56:22.782596 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.782600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.783477 (+ 877us) server_negotiation.cc:407] Connection header received
0929 00:56:22.785291 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.785305 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.785410 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.785525 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:22.787035 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.788029 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.790549 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.790966 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.791060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.794220 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:22.794225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:22.794228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:22.794256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:22.796871 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.797490 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.797493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.797495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.797603 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.797928 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.797931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.797932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.798164 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:22.798264 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.798849 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.798997 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260929 00:56:22.800196 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.782638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33721 (local address 127.0.97.3:56769)
0929 00:56:22.783328 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:22.783365 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:22.784978 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:22.785747 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:22.785755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:22.786223 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:56:22.786883 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.786900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.788683 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.788688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.790431 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.790443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.791145 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.791150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.791335 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.791956 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:56:22.791976 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:56:22.794034 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:56:22.796987 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:22.796993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:22.797004 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:22.797399 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:22.797688 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:56:22.797690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:56:22.797693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:56:22.797844 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:56:22.798261 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:56:22.798265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:56:22.798556 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 00:56:22.798782 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 00:56:22.799125 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
I20260929 00:56:22.801605 12167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33721
I20260929 00:56:22.801942 12167 heartbeater.cc:464] Registering TS with master...
I20260929 00:56:22.802698 12167 heartbeater.cc:511] Master 127.0.97.62:33721 requested a full tablet report, sending...
I20260929 00:56:22.804080 11675 ts_manager.cc:194] Registered new tserver with Master: 902f2a99ae3b4753a0903de627c6109d (127.0.97.3:46339)
I20260929 00:56:22.804929 11675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56769
I20260929 00:56:22.814535 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:56:22.830632 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.819092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.0.97.62:33721)
0929 00:56:22.819251 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.819255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.819777 (+ 522us) server_negotiation.cc:407] Connection header received
0929 00:56:22.819949 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.819953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.820023 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.820105 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:22.821440 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.822141 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.823381 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.823772 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.823859 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.826130 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:22.826135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:22.826138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:22.826164 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:22.828596 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.829165 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.829169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.829171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.829283 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:22.829568 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:22.829571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:22.829573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:22.829797 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:22.829978 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.830247 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.830397 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:56:22.834323 11675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:56:22.841876 11675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:56:22.865552 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.854071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39653 (local address 127.0.0.1:42322)
0929 00:56:22.855080 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:22.855122 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:22.855273 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:22.856465 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:22.856471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:22.856518 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:22.857177 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.857190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.859144 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.859154 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.861746 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.861757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.863835 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.863842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.864039 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.864089 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 00:56:22.864218 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 00:56:22.864412 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":2721,"threads_started":1}
I20260929 00:56:22.865528 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.855094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.97.1:44937)
0929 00:56:22.856188 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.856196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.857381 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 00:56:22.857502 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.857509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.858756 (+ 1247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.858988 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:22.859800 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.861376 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.863154 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.864430 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.864580 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.864800 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.865036 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.865192 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":778,"threads_started":1}
I20260929 00:56:22.866096 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.854624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.0.97.2:39653)
0929 00:56:22.855834 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.855845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.855879 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:56:22.855983 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.855993 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.856257 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.856462 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:22.857648 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.858986 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.861907 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.863410 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.863538 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.863742 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.865610 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.865792 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":950,"threads_started":1}
I20260929 00:56:22.866183 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.854239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44937 (local address 127.0.0.1:41482)
0929 00:56:22.857092 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:22.857120 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:22.857255 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:22.858985 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:22.858989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:22.859015 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:22.859557 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.859569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.861496 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.861501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.862924 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.862934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.864672 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.864678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.864846 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.864876 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:56:22.864994 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 00:56:22.865108 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2715,"thread_start_us":572,"threads_started":1}
I20260929 00:56:22.868122 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.858072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46339 (local address 127.0.0.1:44752)
0929 00:56:22.859767 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:22.859784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:22.859906 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:22.860886 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:22.860891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:22.860927 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:22.861269 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.861280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.863040 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.863046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.865389 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:22.865399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.867390 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.867396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:22.867555 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.867593 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:56:22.867702 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:56:22.867814 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":1371,"threads_started":1}
I20260929 00:56:22.868225 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:22.857608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.0.97.3:46339)
0929 00:56:22.858381 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:22.858401 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:22.859881 (+ 1480us) server_negotiation.cc:407] Connection header received
0929 00:56:22.860048 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:22.860058 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:22.860436 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:22.860638 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:22.861453 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.862890 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.865521 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:22.867195 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:22.867346 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:22.867559 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:22.867739 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 00:56:22.867895 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":554,"threads_started":1}
I20260929 00:56:22.869830 11825 tablet_service.cc:1467] Processing CreateTablet for tablet 373170d34a284c0c94dee7edc5167bdc (DEFAULT_TABLE table=test-table [id=140d8a80658149fe83c8b52bec3d2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:22.871392 11825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373170d34a284c0c94dee7edc5167bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:22.871013 11968 tablet_service.cc:1467] Processing CreateTablet for tablet 373170d34a284c0c94dee7edc5167bdc (DEFAULT_TABLE table=test-table [id=140d8a80658149fe83c8b52bec3d2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:22.871333 12102 tablet_service.cc:1467] Processing CreateTablet for tablet 373170d34a284c0c94dee7edc5167bdc (DEFAULT_TABLE table=test-table [id=140d8a80658149fe83c8b52bec3d2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:56:22.872320 11968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373170d34a284c0c94dee7edc5167bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:22.872571 12102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373170d34a284c0c94dee7edc5167bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:56:22.884972 12186 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Bootstrap starting.
I20260929 00:56:22.888046 12186 tablet_bootstrap.cc:654] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:22.889282 12186 log.cc:824] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:22.890692 12187 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Bootstrap starting.
I20260929 00:56:22.892545 12186 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: No bootstrap required, opened a new log
I20260929 00:56:22.892877 12186 ts_tablet_manager.cc:1397] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 00:56:22.894601 12187 tablet_bootstrap.cc:654] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:22.895761 12187 log.cc:824] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:22.896910 12188 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Bootstrap starting.
I20260929 00:56:22.899132 12187 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: No bootstrap required, opened a new log
I20260929 00:56:22.899406 12187 ts_tablet_manager.cc:1397] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 00:56:22.900072 12188 tablet_bootstrap.cc:654] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Neither blocks nor log segments found. Creating new log.
I20260929 00:56:22.901592 12188 log.cc:824] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Log is configured to *not* fsync() on all Append() calls
I20260929 00:56:22.903949 12186 raft_consensus.cc:348] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.904302 12186 raft_consensus.cc:374] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:22.904423 12186 raft_consensus.cc:729] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902f2a99ae3b4753a0903de627c6109d, State: Initialized, Role: FOLLOWER
I20260929 00:56:22.904740 12188 tablet_bootstrap.cc:492] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: No bootstrap required, opened a new log
I20260929 00:56:22.905031 12188 ts_tablet_manager.cc:1397] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260929 00:56:22.905053 12186 consensus_queue.cc:260] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.906114 12167 heartbeater.cc:503] Master 127.0.97.62:33721 was elected leader, sending a full tablet report...
I20260929 00:56:22.906728 12186 ts_tablet_manager.cc:1428] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 00:56:22.909348 12187 raft_consensus.cc:348] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.909776 12187 raft_consensus.cc:374] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:22.909873 12187 raft_consensus.cc:729] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5c2fc9420f4564954bc59e3670509b, State: Initialized, Role: FOLLOWER
I20260929 00:56:22.910454 12187 consensus_queue.cc:260] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.912130 12187 ts_tablet_manager.cc:1428] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 00:56:22.914794 12188 raft_consensus.cc:348] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.915136 12188 raft_consensus.cc:374] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:56:22.915246 12188 raft_consensus.cc:729] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741076093dae49cb956187da73e792ca, State: Initialized, Role: FOLLOWER
I20260929 00:56:22.915843 12188 consensus_queue.cc:260] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:22.917380 12033 heartbeater.cc:503] Master 127.0.97.62:33721 was elected leader, sending a full tablet report...
I20260929 00:56:22.917845 12188 ts_tablet_manager.cc:1428] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260929 00:56:22.993538 12034 tablet.cc:2369] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:23.008351 12194 raft_consensus.cc:482] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:56:23.008659 12194 raft_consensus.cc:504] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:23.010251 12194 leader_election.cc:288] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed5c2fc9420f4564954bc59e3670509b (127.0.97.1:44937), 902f2a99ae3b4753a0903de627c6109d (127.0.97.3:46339)
W20260929 00:56:23.018458 11891 tablet.cc:2369] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:23.019603 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.010644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38097 (local address 127.0.97.3:46339)
0929 00:56:23.010851 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.010858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.011620 (+ 762us) server_negotiation.cc:407] Connection header received
0929 00:56:23.011797 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.011801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.011867 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.011959 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:23.013027 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.014159 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.017262 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.018496 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.018632 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.018735 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.019282 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.019419 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:56:23.019639 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.010746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39079 (local address 127.0.97.1:44937)
0929 00:56:23.010935 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.010943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.011568 (+ 625us) server_negotiation.cc:407] Connection header received
0929 00:56:23.011769 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.011774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.011855 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.011959 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:56:23.013027 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.014159 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.017262 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.018506 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.018634 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.018735 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.019254 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.019379 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:56:23.019771 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.010628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44937 (local address 127.0.97.2:39079)
0929 00:56:23.011460 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:23.011475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:23.011606 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:23.012001 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:23.012006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:23.012043 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:23.012837 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:23.012859 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.014317 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.014323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:23.017056 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:23.017071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.018879 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.018885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:23.019069 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.019119 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 00:56:23.019228 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:56:23.019359 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":217,"spinlock_wait_cycles":3968,"thread_start_us":411,"threads_started":1}
I20260929 00:56:23.020285 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.010522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46339 (local address 127.0.97.2:38097)
0929 00:56:23.011437 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 00:56:23.011461 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:56:23.011647 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:56:23.012001 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:56:23.012006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:56:23.012038 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:56:23.012898 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:23.012906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.014319 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.014323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:23.017056 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:56:23.017071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.018838 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.018846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:56:23.019064 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.019115 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 00:56:23.019791 (+ 676us) client_negotiation.cc:240] Negotiation successful
0929 00:56:23.019906 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":276,"spinlock_wait_cycles":73600,"thread_start_us":588,"threads_started":1}
I20260929 00:56:23.020819 11845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373170d34a284c0c94dee7edc5167bdc" candidate_uuid: "741076093dae49cb956187da73e792ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5c2fc9420f4564954bc59e3670509b" is_pre_election: true
I20260929 00:56:23.021262 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373170d34a284c0c94dee7edc5167bdc" candidate_uuid: "741076093dae49cb956187da73e792ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902f2a99ae3b4753a0903de627c6109d" is_pre_election: true
I20260929 00:56:23.021430 11845 raft_consensus.cc:2454] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741076093dae49cb956187da73e792ca in term 0.
I20260929 00:56:23.021809 12122 raft_consensus.cc:2454] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741076093dae49cb956187da73e792ca in term 0.
I20260929 00:56:23.022142 11924 leader_election.cc:302] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741076093dae49cb956187da73e792ca, ed5c2fc9420f4564954bc59e3670509b; no voters:
I20260929 00:56:23.022543 12194 raft_consensus.cc:2781] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:56:23.022709 12194 raft_consensus.cc:482] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:56:23.022831 12194 raft_consensus.cc:3037] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:23.025369 12194 raft_consensus.cc:504] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:23.025908 12194 leader_election.cc:288] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [CANDIDATE]: Term 1 election: Requested vote from peers ed5c2fc9420f4564954bc59e3670509b (127.0.97.1:44937), 902f2a99ae3b4753a0903de627c6109d (127.0.97.3:46339)
I20260929 00:56:23.026343 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373170d34a284c0c94dee7edc5167bdc" candidate_uuid: "741076093dae49cb956187da73e792ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902f2a99ae3b4753a0903de627c6109d"
I20260929 00:56:23.026358 11845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373170d34a284c0c94dee7edc5167bdc" candidate_uuid: "741076093dae49cb956187da73e792ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5c2fc9420f4564954bc59e3670509b"
I20260929 00:56:23.026544 11845 raft_consensus.cc:3037] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:23.026539 12122 raft_consensus.cc:3037] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:56:23.029618 11845 raft_consensus.cc:2454] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741076093dae49cb956187da73e792ca in term 1.
I20260929 00:56:23.029618 12122 raft_consensus.cc:2454] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741076093dae49cb956187da73e792ca in term 1.
I20260929 00:56:23.030014 11924 leader_election.cc:302] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 741076093dae49cb956187da73e792ca, 902f2a99ae3b4753a0903de627c6109d, ed5c2fc9420f4564954bc59e3670509b; no voters:
I20260929 00:56:23.030319 12194 raft_consensus.cc:2781] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:56:23.030762 12194 raft_consensus.cc:686] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 1 LEADER]: Becoming Leader. State: Replica: 741076093dae49cb956187da73e792ca, State: Running, Role: LEADER
W20260929 00:56:23.031350 12168 tablet.cc:2369] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:56:23.031256 12194 consensus_queue.cc:237] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } }
I20260929 00:56:23.036010 11676 catalog_manager.cc:5495] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 741076093dae49cb956187da73e792ca (127.0.97.2). New cstate: current_term: 1 leader_uuid: "741076093dae49cb956187da73e792ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741076093dae49cb956187da73e792ca" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 } health_report { overall_health: UNKNOWN } } }
I20260929 00:56:23.053946 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.048632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.0.97.2:39653)
0929 00:56:23.048815 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.048819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.048835 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:56:23.048937 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.048940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.049009 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.049124 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:56:23.049445 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.050139 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.052090 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.052469 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.052615 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.053021 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:56:23.053484 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:56:23.053594 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.053681 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.053739 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:56:23.062105 11845 raft_consensus.cc:1261] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 1 FOLLOWER]: Refusing update from remote peer 741076093dae49cb956187da73e792ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:23.062105 12122 raft_consensus.cc:1261] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 1 FOLLOWER]: Refusing update from remote peer 741076093dae49cb956187da73e792ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:56:23.063223 12194 consensus_queue.cc:1035] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5c2fc9420f4564954bc59e3670509b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:23.063544 12197 consensus_queue.cc:1035] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "902f2a99ae3b4753a0903de627c6109d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:56:23.076313 12202 mvcc.cc:205] Tried to move back new op lower bound from 7334475297011159040 to 7334475296900263936. Current Snapshot: MvccSnapshot[applied={T|T < 7334475297011159040}]
I20260929 00:56:23.090461 11676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:23.090747 11676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:56:23.094197 11676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00029d120f1048bc94d3720aa2d8dae5: Sending DeleteTablet for 3 replicas of tablet 373170d34a284c0c94dee7edc5167bdc
I20260929 00:56:23.095355 11825 tablet_service.cc:1514] Processing DeleteTablet for tablet 373170d34a284c0c94dee7edc5167bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41482
I20260929 00:56:23.095355 12102 tablet_service.cc:1514] Processing DeleteTablet for tablet 373170d34a284c0c94dee7edc5167bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44752
I20260929 00:56:23.095700 11968 tablet_service.cc:1514] Processing DeleteTablet for tablet 373170d34a284c0c94dee7edc5167bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:56:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260929 00:56:23.098119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 11757
I20260929 00:56:23.129819 12211 tablet_replica.cc:330] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: stopping tablet replica
I20260929 00:56:23.129976 12209 tablet_replica.cc:330] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: stopping tablet replica
I20260929 00:56:23.130329 12211 raft_consensus.cc:2229] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:56:23.130411 12209 raft_consensus.cc:2229] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:23.130941 12209 raft_consensus.cc:2258] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:23.130956 12211 raft_consensus.cc:2258] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:23.132326 12210 tablet_replica.cc:330] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: stopping tablet replica
I20260929 00:56:23.132738 12210 raft_consensus.cc:2229] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:56:23.133210 12210 raft_consensus.cc:2258] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:56:23.133572 12209 ts_tablet_manager.cc:1905] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:23.135242 12210 ts_tablet_manager.cc:1905] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:56:23 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:56:23.141520 12209 ts_tablet_manager.cc:1918] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:23.141682 12209 log.cc:1196] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/373170d34a284c0c94dee7edc5167bdc
I20260929 00:56:23.142009 12210 ts_tablet_manager.cc:1918] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:23.142168 12210 log.cc:1196] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/373170d34a284c0c94dee7edc5167bdc
I20260929 00:56:23.142251 12209 ts_tablet_manager.cc:1939] T 373170d34a284c0c94dee7edc5167bdc P ed5c2fc9420f4564954bc59e3670509b: Deleting consensus metadata
I20260929 00:56:23.142663 12210 ts_tablet_manager.cc:1939] T 373170d34a284c0c94dee7edc5167bdc P 902f2a99ae3b4753a0903de627c6109d: Deleting consensus metadata
I20260929 00:56:23.144338 11662 catalog_manager.cc:4841] TS 902f2a99ae3b4753a0903de627c6109d (127.0.97.3:46339): tablet 373170d34a284c0c94dee7edc5167bdc (table test-table [id=140d8a80658149fe83c8b52bec3d2cb7]) successfully deleted
I20260929 00:56:23.144652 11663 catalog_manager.cc:4841] TS ed5c2fc9420f4564954bc59e3670509b (127.0.97.1:44937): tablet 373170d34a284c0c94dee7edc5167bdc (table test-table [id=140d8a80658149fe83c8b52bec3d2cb7]) successfully deleted
I20260929 00:56:23.146849 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.099950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.97.1:44937)
0929 00:56:23.100169 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.100178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.121208 (+ 21030us) server_negotiation.cc:407] Connection header received
0929 00:56:23.121781 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.121790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.121921 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.122026 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:23.124335 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.125994 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.131528 (+ 5534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.132168 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.132305 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.138123 (+ 5818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:23.138140 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:23.138157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:23.138204 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:23.142691 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.143800 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.143810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.143814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.143993 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.145116 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.145121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.145124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.145499 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:23.145707 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.146217 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.146475 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:56:23.147223 12211 ts_tablet_manager.cc:1905] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:56:23.147558 11885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:56:23.154654 12211 ts_tablet_manager.cc:1918] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:56:23.154847 12211 log.cc:1196] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/373170d34a284c0c94dee7edc5167bdc
I20260929 00:56:23.155351 12211 ts_tablet_manager.cc:1939] T 373170d34a284c0c94dee7edc5167bdc P 741076093dae49cb956187da73e792ca: Deleting consensus metadata
I20260929 00:56:23.157176 11662 catalog_manager.cc:4841] TS 741076093dae49cb956187da73e792ca (127.0.97.2:39653): tablet 373170d34a284c0c94dee7edc5167bdc (table test-table [id=140d8a80658149fe83c8b52bec3d2cb7]) successfully deleted
I20260929 00:56:23.249336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 11757
I20260929 00:56:23.268110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 11896
Sep 29 00:56:23 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:56:23.285557 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.268908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.97.2:39653)
0929 00:56:23.269082 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.269086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.269131 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:56:23.269361 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.269364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.269424 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.269514 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:23.271187 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.272012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.273965 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.274355 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.274450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.279275 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:23.279287 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:23.279304 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:23.279338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:23.282763 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.283417 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.283432 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.283435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.283577 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.283958 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.283963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.283966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.284316 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:23.284678 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.284953 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.285170 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:56:23.286379 12028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:56:23.398914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 11896
I20260929 00:56:23.416996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 12039
Sep 29 00:56:23 dist-test-slave-pzsx krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:56:23.431401 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:56:23.417782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.97.3:46339)
0929 00:56:23.417933 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:56:23.417937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:56:23.417968 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:56:23.418127 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:56:23.418131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:56:23.418192 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:56:23.418272 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:56:23.419515 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.420195 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.421530 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:56:23.421897 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:56:23.421989 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:56:23.426163 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:56:23.426176 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:56:23.426192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:56:23.426226 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:56:23.428867 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.429656 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.429663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.429665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.429787 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:56:23.430146 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:56:23.430149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:56:23.430151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:56:23.430487 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:56:23.430688 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 00:56:23.430971 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:56:23.431115 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:56:23.431965 12162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:56:23.526670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 12039
I20260929 00:56:23.542879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 11644
I20260929 00:56:23.544034 11706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:56:23.653223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 11644
I20260929 00:56:23.678171 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 00:56:24.014008 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 00:56:24.014241 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:56:24.360853 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:56:24.361 UTC [11230] LOG: received smart shutdown request
2026-09-29 00:56:24.364 UTC [11230] LOG: background worker "logical replication launcher" (PID 11239) exited with exit code 1
2026-09-29 00:56:24.365 UTC [11234] LOG: shutting down
2026-09-29 00:56:24.531 UTC [11230] LOG: database system is shut down
2026-09-29T00:56:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47815 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:56:24 dist-test-slave-pzsx krb5kdc[12225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:56:24 dist-test-slave-pzsx krb5kdc[12225](info): set up 2 sockets
Sep 29 00:56:24 dist-test-slave-pzsx krb5kdc[12225](info): commencing operation
krb5kdc: starting...
W20260929 00:56:25.748798 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:56:25 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:56:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 00:56:26.055402 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiPz2y/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/postgres -l logfile start
2026-09-29 00:56:28.532 UTC [12255] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 00:56:28.532 UTC [12255] LOG: listening on IPv4 address "127.0.97.20", port 33687
2026-09-29 00:56:28.535 UTC [12255] LOG: listening on Unix socket "/tmp/.s.PGSQL.33687"
2026-09-29 00:56:28.558 UTC [12258] LOG: database system was shut down at 2026-09-29 00:56:27 UTC
2026-09-29 00:56:28.561 UTC [12255] LOG: database system is ready to accept connections
I20260929 00:56:29.524471 388 mini_postgres.cc:96] Postgres bound to 33687
2026-09-29 00:56:29.540 UTC [12266] FATAL: database "slave" does not exist
127.0.97.20:33687 - accepting connections
I20260929 00:56:29.541559 388 mini_ranger.cc:161] Starting Ranger...
I20260929 00:56:29.565573 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 00:56:30.021241 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 00:56:30.021361 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin
2026-09-29 00:56:30,311 [I] DB FLAVOR :POSTGRES
2026-09-29 00:56:30,311 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:56:30,311 [I] Checking connection..
2026-09-29 00:56:30,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 00:56:30,558 [I] Checking connection passed.
2026-09-29 00:56:30,558 [I] --------- Verifying version history table ---------
2026-09-29 00:56:30,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:56:30,809 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:56:30,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:56:31,058 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 00:56:31,058 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 00:56:31,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 00:56:31.289 UTC [12330] WARNING: there is no transaction in progress
2026-09-29 00:56:31,295 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 00:56:31,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 00:56:31,535 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 00:56:31,535 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:56:31,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 00:56:31,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 00:56:32,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pzsx.c.gcp-upstream.internal','N') ;"
2026-09-29 00:56:32,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:56:32,541 [I] Table x_portal_user does not exist in database ranger
2026-09-29 00:56:32,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:56:32,785 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 00:56:32,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:56:33,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:56:33,265 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 00:56:33,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 00:56:34.186 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.202 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.218 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.233 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.250 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.268 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.332 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.339 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.354 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.363 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.373 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.386 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.396 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.402 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.408 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:34.415 UTC [12438] WARNING: there is no transaction in progress
2026-09-29 00:56:35,012 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 00:56:35,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 00:56:35,268 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 00:56:35,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 00:56:35,523 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 00:56:35,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 00:56:35,751 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 00:56:35,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 00:56:35,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 00:56:36,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pzsx.c.gcp-upstream.internal';"
2026-09-29 00:56:36,483 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 00:56:36,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 00:56:36,742 [I] DB_PATCHES have already been applied
I20260929 00:56:36.748423 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiPz2y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 00:56:36.748559 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 00:56:36.763482 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:45929
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 12:56:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 00:56:37.879915 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.338s user 0.003s sys 0.034s
Sep 29, 2026 12:56:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 12:56:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 12:56:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 12:56:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 12:56:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 00:56:57.932132 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 00:56:58 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643418, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 00:56:58.102243 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 00:56:58.122610 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 00:56:58.962142 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 00:56:58.962260 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms
I20260929 00:56:58.962319 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33687
2026-09-29 00:56:59,322 [I] DB FLAVOR :POSTGRES
2026-09-29 00:56:59,323 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 00:56:59,323 [I] Checking connection
2026-09-29 00:56:59,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 00:56:59,576 [I] connection success
2026-09-29 00:56:59,577 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 00:56:59,577 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 00:56:59,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 00:56:59,826 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 00:56:59,827 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 00:56:59,827 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 00:56:59,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33687/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 00:57:00,085 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 00:57:00.572154 388 mini_ranger_kms.cc:310] Created kms service
I20260929 00:57:00.736749 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 00:57:00.783095 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 00:57:00 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643418, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:01.192734 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 00:57:01.193080 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiPz2y/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiPz2y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 00:57:01.193145 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 00:57:01.207506 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12618
I20260929 00:57:01.207639 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:48165
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 12:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 00:57:03.472496 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.370s user 0.063s sys 0.308s
I20260929 00:57:03.472774 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 00:57:03.472944 388 mini_ranger_kms.cc:401] 127.0.97.20:48165/kms/v1/keys
Sep 29 00:57:03 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 00:57:04 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 00:57:04 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 00:57:06 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:57:06.739154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44457
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44457
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:07.069590 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:07.069852 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:07.070032 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:07.077060 12669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:07.077162 12669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:07.077252 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:07.077351 12669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:07.077440 12669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:07.087283 12669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:48165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44457
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:07.088572 12669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:07.089900 12669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:07.102423 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:07.102502 12674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:07.102530 12675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:07.103410 12669 server_base.cc:1034] running on GCE node
I20260929 00:57:07.104131 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:07.105518 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:07.106734 12669 hybrid_clock.cc:648] HybridClock initialized: now 1790643427106708 us; error 36 us; skew 500 ppm
I20260929 00:57:07.107064 12669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:07 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:07.117661 12669 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:07.119019 12669 webserver.cc:460] Webserver started at http://127.0.97.62:34693/ using document root <none> and password file <none>
I20260929 00:57:07.119582 12669 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:07.119666 12669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:07.119885 12669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:57:07 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:07.166947 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "01e016cfc83a4342b0d7517f1e78ff40"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-pzsx"
server_key: "c8a800ea7cd0707e03d7bc5364f44384"
server_key_iv: "7cde5861f2613c79c7efefcd1f83d10c"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:07.167795 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "01e016cfc83a4342b0d7517f1e78ff40"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-pzsx"
server_key: "c8a800ea7cd0707e03d7bc5364f44384"
server_key_iv: "7cde5861f2613c79c7efefcd1f83d10c"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:07.173250 12669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:57:07.192209 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:07.193764 12669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:57:07.193974 12669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "01e016cfc83a4342b0d7517f1e78ff40"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-pzsx"
server_key: "c8a800ea7cd0707e03d7bc5364f44384"
server_key_iv: "7cde5861f2613c79c7efefcd1f83d10c"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:07.194193 12669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:07.205803 12669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:07.210335 12669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:07.210690 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:07.230610 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44457 every 8 connection(s)
I20260929 00:57:07.230609 12669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44457
I20260929 00:57:07.232419 12669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:07.233189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 12669
I20260929 00:57:07.233418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:57:07.234277 388 external_mini_cluster.cc:1428] Setting key e2822ac056fa5a5429fd96794ede69ae
I20260929 00:57:07.237607 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:07.254146 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Bootstrap starting.
I20260929 00:57:07.257578 12736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:07.258775 12736 log.cc:824] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:57:07 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:07.261871 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: No bootstrap required, opened a new log
I20260929 00:57:07.271646 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:07.242280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.97.62:44457)
0929 00:57:07.247451 (+ 5171us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:07.247497 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:07.247547 (+ 50us) server_negotiation.cc:407] Connection header received
0929 00:57:07.249984 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:07.250015 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:07.250540 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:07.250999 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:07.252724 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.254649 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.256559 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:07.257595 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:07.257763 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:07.262738 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:07.262748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:07.262766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:07.262812 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:07.266987 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:07.267790 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:07.267799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:07.267813 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:07.267992 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:07.268420 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:07.268425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:07.268429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:07.269135 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:07.269475 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:07.269891 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 00:57:07.270315 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4937,"thread_start_us":4726,"threads_started":1}
I20260929 00:57:07.272428 12736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:07.272858 12736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:07.272967 12736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e016cfc83a4342b0d7517f1e78ff40, State: Initialized, Role: FOLLOWER
I20260929 00:57:07.273578 12736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:07.273842 12736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:07.273983 12736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:07.274168 12736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:07.276391 12736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:07.276888 12736 leader_election.cc:302] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e016cfc83a4342b0d7517f1e78ff40; no voters:
I20260929 00:57:07.277518 12736 leader_election.cc:288] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:07.277662 12741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:07.281579 12741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 LEADER]: Becoming Leader. State: Replica: 01e016cfc83a4342b0d7517f1e78ff40, State: Running, Role: LEADER
I20260929 00:57:07.282241 12741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:07.283372 12736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:07.285910 12742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e016cfc83a4342b0d7517f1e78ff40. Latest consensus state: current_term: 1 leader_uuid: "01e016cfc83a4342b0d7517f1e78ff40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } } }
I20260929 00:57:07.286264 12742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:07.286787 12743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e016cfc83a4342b0d7517f1e78ff40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } } }
I20260929 00:57:07.287041 12743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:07.291211 12746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:07.299762 12746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:07.321466 12746 catalog_manager.cc:1347] Generated new cluster ID: 37943dec284b4e08a534ed53f2252f4e
I20260929 00:57:07.321594 12746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:07.339752 12746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:07.341640 12746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:07.352095 12746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Generated new TSK 0
I20260929 00:57:07.353078 12746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:07.356177 12736 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:57:08.176472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:08 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 00:57:08.588801 12781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:08.589112 12781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:08.589291 12781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:08.596846 12781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:08.596982 12781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:08.597162 12781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:08.607782 12781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:08.609287 12781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:08.610572 12781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:08.627929 12792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:08.628315 12791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:08.631312 12794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:08.632072 12781 server_base.cc:1034] running on GCE node
I20260929 00:57:08.632772 12781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:08.633936 12781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:08.635165 12781 hybrid_clock.cc:648] HybridClock initialized: now 1790643428635097 us; error 57 us; skew 500 ppm
I20260929 00:57:08.635490 12781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:08 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:08.638901 12781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:08.645913 12781 webserver.cc:460] Webserver started at http://127.0.97.1:45913/ using document root <none> and password file <none>
I20260929 00:57:08.646932 12781 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:08.647071 12781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:08.647428 12781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:57:08 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:08.668324 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "0476223715af4950ab895cb170cb9521"
format_stamp: "Formatted at 2026-09-29 00:57:08 on dist-test-slave-pzsx"
server_key: "5d72940ba9d8ad86d60afdf00756d9e3"
server_key_iv: "e2e228e56ac97cf7ae58b81144587045"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:08.669497 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0476223715af4950ab895cb170cb9521"
format_stamp: "Formatted at 2026-09-29 00:57:08 on dist-test-slave-pzsx"
server_key: "5d72940ba9d8ad86d60afdf00756d9e3"
server_key_iv: "e2e228e56ac97cf7ae58b81144587045"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:08.676844 12781 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 00:57:08.705190 12801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:08.707332 12781 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260929 00:57:08.707552 12781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "0476223715af4950ab895cb170cb9521"
format_stamp: "Formatted at 2026-09-29 00:57:08 on dist-test-slave-pzsx"
server_key: "5d72940ba9d8ad86d60afdf00756d9e3"
server_key_iv: "e2e228e56ac97cf7ae58b81144587045"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:08.707837 12781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:08.735540 12781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:08.741070 12781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:08.741418 12781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:08.742661 12781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:08.744519 12781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:08.744632 12781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 00:57:08.744786 12781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:08.744885 12781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:08.823202 12781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34793
I20260929 00:57:08.823222 12913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34793 every 8 connection(s)
I20260929 00:57:08.824903 12781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:08.832791 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 12781
I20260929 00:57:08.833047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:08.834022 388 external_mini_cluster.cc:1428] Setting key 7758be2183f287acfc20d7da2d7cf3c9
Sep 29 00:57:08 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:57:08.866674 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:08.827722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41869 (local address 127.0.97.62:44457)
0929 00:57:08.841385 (+ 13663us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:08.841391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:08.841411 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:57:08.841471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:08.841477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:08.841563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:08.841677 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:08.843830 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:08.845075 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:08.848206 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:08.848913 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:08.849045 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:08.858390 (+ 9345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:08.858398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:08.858404 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:08.858450 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:08.862462 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:08.863485 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:08.863491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:08.863495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:08.863748 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:08.864321 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:08.864326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:08.864330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:08.864719 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:08.864857 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:08.865999 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0929 00:57:08.866231 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13510,"thread_start_us":16765,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 00:57:08.869177 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:08.828337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.1:41869)
0929 00:57:08.837268 (+ 8931us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:08.837340 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:08.839785 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:08.842130 (+ 2345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:08.842143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:08.842791 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:08.843613 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:08.843640 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:08.845905 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:08.845910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:08.848037 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:08.848053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:08.851258 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:08.851266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:08.851567 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:08.852690 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:08.852726 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:08.857947 (+ 5221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:08.862659 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:08.862670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:08.862691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:08.863336 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:08.863910 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:08.863915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:08.863919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:08.864186 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:08.864976 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:08.864985 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:08.865475 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 00:57:08.867004 (+ 1529us) client_negotiation.cc:240] Negotiation successful
0929 00:57:08.867569 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8601,"thread_start_us":11755,"threads_started":1}
I20260929 00:57:08.871124 12914 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:08.871505 12914 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:08.872355 12914 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:08.874852 12701 ts_manager.cc:194] Registered new tserver with Master: 0476223715af4950ab895cb170cb9521 (127.0.97.1:34793)
I20260929 00:57:08.876958 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:57:08.904035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:08 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 00:57:09 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 00:57:09.403358 12920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:09.403862 12920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:09.404052 12920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:09.415498 12920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:09.415719 12920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:09.415949 12920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:09.432041 12920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:09.434041 12920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:09.435819 12920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:09.477134 12928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.480208 12931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:09.480732 12927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:09.481247 12920 server_base.cc:1034] running on GCE node
I20260929 00:57:09.482184 12920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:09.487512 12920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:09.488924 12920 hybrid_clock.cc:648] HybridClock initialized: now 1790643429488800 us; error 117 us; skew 500 ppm
I20260929 00:57:09.489414 12920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:09 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:09.495359 12920 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:09.546531 12920 webserver.cc:460] Webserver started at http://127.0.97.2:33085/ using document root <none> and password file <none>
I20260929 00:57:09.547773 12920 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:09.548012 12920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:09.548456 12920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:57:09 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:09.581935 12920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "0f79ee68802e4f7abb681c8864afa440"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-pzsx"
server_key: "37f3a6f86b35a1d91d9f98abb2a2b560"
server_key_iv: "1f541b2f22a4828e1f97c65ac178dcd6"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:09.583099 12920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0f79ee68802e4f7abb681c8864afa440"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-pzsx"
server_key: "37f3a6f86b35a1d91d9f98abb2a2b560"
server_key_iv: "1f541b2f22a4828e1f97c65ac178dcd6"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:09.596490 12920 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260929 00:57:09.634943 12937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.636842 12920 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:57:09.637125 12920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "0f79ee68802e4f7abb681c8864afa440"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-pzsx"
server_key: "37f3a6f86b35a1d91d9f98abb2a2b560"
server_key_iv: "1f541b2f22a4828e1f97c65ac178dcd6"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:09.637496 12920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:09.709474 12920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:09.714488 12920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:09.714789 12920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:09.715927 12920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:09.717818 12920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:09.717918 12920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.718043 12920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:09.718106 12920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:09.751083 12920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44723
I20260929 00:57:09.751103 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44723 every 8 connection(s)
I20260929 00:57:09.752770 12920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:09.753556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 12920
I20260929 00:57:09.753746 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:09.754345 388 external_mini_cluster.cc:1428] Setting key 1dd98cd2411f8bf337b5b28198889f4a
Sep 29 00:57:09 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:09.783223 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.755585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59855 (local address 127.0.97.62:44457)
0929 00:57:09.760915 (+ 5330us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:09.760921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:09.760939 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:57:09.764596 (+ 3657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:09.764606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:09.764721 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:09.764854 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:09.766822 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.768056 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.770468 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.771240 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.771400 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.775423 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:09.775432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:09.775440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:09.775501 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:09.779455 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.780344 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.780351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.780355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.780545 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:09.781052 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:09.781060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:09.781065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:09.781501 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:09.781657 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:09.782529 (+ 872us) server_negotiation.cc:299] Negotiation successful
0929 00:57:09.782787 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5208,"thread_start_us":5107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:57:09.784235 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:09.756446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.2:59855)
0929 00:57:09.758926 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:09.758975 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:09.761428 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:09.765150 (+ 3722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:09.765162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:09.765800 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:09.766642 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.766667 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.768233 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.768238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.770308 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:09.770323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:09.771495 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:09.771502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:09.771756 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:09.772559 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:09.772587 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:09.775191 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:09.779613 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.779623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.779640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.780195 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.780653 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:09.780658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:09.780661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:09.780903 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:09.781639 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:09.781647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:09.782051 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 00:57:09.782359 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 00:57:09.782876 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2114,"thread_start_us":1880,"threads_started":1}
I20260929 00:57:09.786036 13057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:09.786406 13057 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:09.787213 13057 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 00:57:09.788864 12701 ts_manager.cc:194] Registered new tserver with Master: 0f79ee68802e4f7abb681c8864afa440 (127.0.97.2:44723)
I20260929 00:57:09.790123 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:57:09.832036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:09.880551 12914 heartbeater.cc:503] Master 127.0.97.62:44457 was elected leader, sending a full tablet report...
W20260929 00:57:10.160550 13063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:10.160864 13063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:10.161041 13063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:10.168228 13063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:10.168373 13063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:10.168596 13063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:10.178318 13063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:10.179627 13063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:10.181137 13063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:10.193792 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:10.193984 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:10.194042 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:10.194876 13063 server_base.cc:1034] running on GCE node
I20260929 00:57:10.195370 13063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:10.196198 13063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:10.197392 13063 hybrid_clock.cc:648] HybridClock initialized: now 1790643430197352 us; error 30 us; skew 500 ppm
I20260929 00:57:10.197707 13063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:10.200783 13063 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:10.202248 13063 webserver.cc:460] Webserver started at http://127.0.97.3:33075/ using document root <none> and password file <none>
I20260929 00:57:10.202939 13063 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:10.203080 13063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:10.203414 13063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:10.220710 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "5b82665cca5f44c7a4bcc701a62308e1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-pzsx"
server_key: "dfefa5ba5ab7babe60f212dd3b10a680"
server_key_iv: "23d652b3aa1a174b651f4bae1a85bbdc"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:10.221892 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5b82665cca5f44c7a4bcc701a62308e1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-pzsx"
server_key: "dfefa5ba5ab7babe60f212dd3b10a680"
server_key_iv: "23d652b3aa1a174b651f4bae1a85bbdc"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:10.228825 13063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 00:57:10.245966 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.247529 13063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:10.247718 13063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "5b82665cca5f44c7a4bcc701a62308e1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-pzsx"
server_key: "dfefa5ba5ab7babe60f212dd3b10a680"
server_key_iv: "23d652b3aa1a174b651f4bae1a85bbdc"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:10.247998 13063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:10.265970 13063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:10.270260 13063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:10.270577 13063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:10.271577 13063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:10.272964 13063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:10.273062 13063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.273204 13063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:10.273279 13063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:10.314642 13063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39557
I20260929 00:57:10.314677 13190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39557 every 8 connection(s)
I20260929 00:57:10.315963 13063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:10.319435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13063
I20260929 00:57:10.319617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:10.320149 388 external_mini_cluster.cc:1428] Setting key f5c58f90709d90944ad838f7113a8caa
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:10.335232 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.318458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37177 (local address 127.0.97.62:44457)
0929 00:57:10.318968 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.318972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.319992 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 00:57:10.322269 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.322275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.322364 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.322458 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.323946 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.324727 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.326163 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.326580 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.326669 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.329839 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.329844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.329847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.329877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.332205 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.332976 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.332983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.332987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.333109 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.333525 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.333530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.333534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.333792 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.333941 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.334653 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.334869 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
I20260929 00:57:10.336148 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.318989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.3:37177)
0929 00:57:10.319828 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:10.319874 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:10.322016 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:10.322721 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:10.322734 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:10.323256 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:10.323806 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:10.323822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.324859 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.324863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:10.326041 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:10.326059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.326752 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.326757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:10.326956 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.327628 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:10.327649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:10.329630 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:10.332354 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:10.332363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:10.332379 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:10.332855 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:10.333215 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:10.333220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:10.333224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:10.333412 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:10.333914 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:10.333922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:10.334314 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 00:57:10.334568 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 00:57:10.335036 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":396,"threads_started":1}
I20260929 00:57:10.337369 13191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:10.337667 13191 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:10.338376 13191 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:10.339771 12701 ts_manager.cc:194] Registered new tserver with Master: 5b82665cca5f44c7a4bcc701a62308e1 (127.0.97.3:39557)
I20260929 00:57:10.340634 12701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37177
I20260929 00:57:10.347491 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:10.349220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 12781
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:57:10.364712 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.350075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.0.97.1:34793)
0929 00:57:10.350593 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.350599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.350623 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:57:10.350680 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.350686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.350863 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.350989 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.352570 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.353563 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.354968 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.355478 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.355570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.359725 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.359737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.359750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.359782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.362391 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.362983 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.362988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.362991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.363113 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.363413 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.363416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.363418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.363723 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.363944 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.364176 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.364363 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260929 00:57:10.365902 12909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:10.447944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 12781
I20260929 00:57:10.465562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 12920
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:57:10.481590 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.466401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.0.97.2:44723)
0929 00:57:10.466984 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.466991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.467017 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:57:10.467094 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.467101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.467307 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.467437 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.468611 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.469827 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.471177 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.471709 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.471805 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.475948 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.475959 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.475974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.476007 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.478638 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.479358 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.479366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.479369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.479514 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.479894 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.479900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.479903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.480253 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.480626 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.480946 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.481218 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":324,"threads_started":1}
I20260929 00:57:10.482887 13052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:10.563992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 12920
I20260929 00:57:10.579658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13063
Sep 29 00:57:10 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:57:10.595003 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:10.580668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.97.3:39557)
0929 00:57:10.581312 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:10.581319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:10.581348 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:57:10.581404 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:10.581410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:10.581636 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:10.581761 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:10.582989 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.584076 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.585328 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:10.585857 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:10.585952 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:10.590013 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:10.590023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:10.590037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:10.590066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:10.592632 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.593251 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.593257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.593259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.593380 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:10.593686 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:10.593689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:10.593691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:10.593993 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:10.594232 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:10.594406 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 00:57:10.594593 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":356,"threads_started":1}
I20260929 00:57:10.596237 13186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:10.677598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13063
I20260929 00:57:10.693245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 12669
I20260929 00:57:10.694474 12731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:10.799738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 12669
I20260929 00:57:10.820794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44457
--webserver_interface=127.0.97.62
--webserver_port=34693
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44457
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:11.136230 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:11.136595 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:11.136780 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:11.143898 13203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:11.143999 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:11.144078 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:11.144178 13203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:11.144268 13203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:11.154495 13203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44457
--ranger_config_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:11.155776 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:11.157099 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:11.168725 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.168637 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.168673 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:11.169515 13203 server_base.cc:1034] running on GCE node
I20260929 00:57:11.170202 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:11.171573 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:11.172793 13203 hybrid_clock.cc:648] HybridClock initialized: now 1790643431172761 us; error 45 us; skew 500 ppm
I20260929 00:57:11.173115 13203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:11 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:11.176072 13203 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:11.177426 13203 webserver.cc:460] Webserver started at http://127.0.97.62:34693/ using document root <none> and password file <none>
I20260929 00:57:11.178061 13203 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:11.178216 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:57:11 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:11.231626 13203 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:57:11.234712 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.236152 13203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:57:11.236363 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "01e016cfc83a4342b0d7517f1e78ff40"
format_stamp: "Formatted at 2026-09-29 00:57:07 on dist-test-slave-pzsx"
server_key: "c8a800ea7cd0707e03d7bc5364f44384"
server_key_iv: "7cde5861f2613c79c7efefcd1f83d10c"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:11.237376 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:11.252301 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:11.255882 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:11.256170 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:11.274526 13203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44457
I20260929 00:57:11.274549 13270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44457 every 8 connection(s)
I20260929 00:57:11.276002 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:11.282860 13271 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:11.283079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13203
I20260929 00:57:11.284112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34793
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44457
--webserver_port=45913
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:11.288097 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Bootstrap starting.
I20260929 00:57:11.308265 13271 log.cc:824] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:11.317262 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:11.317793 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Bootstrap complete.
I20260929 00:57:11.326674 13271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:11.327574 13271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e016cfc83a4342b0d7517f1e78ff40, State: Initialized, Role: FOLLOWER
I20260929 00:57:11.328238 13271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:11.328478 13271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:11.328670 13271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:11.328866 13271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:57:11.332731 13271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:11.333201 13271 leader_election.cc:302] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e016cfc83a4342b0d7517f1e78ff40; no voters:
I20260929 00:57:11.333864 13271 leader_election.cc:288] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:57:11.334000 13275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:57:11.335318 13275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [term 2 LEADER]: Becoming Leader. State: Replica: 01e016cfc83a4342b0d7517f1e78ff40, State: Running, Role: LEADER
I20260929 00:57:11.335922 13275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } }
I20260929 00:57:11.336150 13271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:11.338568 13276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "01e016cfc83a4342b0d7517f1e78ff40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } } }
I20260929 00:57:11.338999 13276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:11.341331 13277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e016cfc83a4342b0d7517f1e78ff40. Latest consensus state: current_term: 2 leader_uuid: "01e016cfc83a4342b0d7517f1e78ff40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e016cfc83a4342b0d7517f1e78ff40" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44457 } } }
I20260929 00:57:11.341661 13277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:11.343369 13281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:11.350409 13281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:11.352954 13281 catalog_manager.cc:1259] Loaded cluster ID: 37943dec284b4e08a534ed53f2252f4e
I20260929 00:57:11.353129 13281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:11.357223 13281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:11.358600 13281 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Loaded TSK: 0
I20260929 00:57:11.360101 13281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:11.362758 13271 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 00:57:11.679768 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:11.680265 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:11.680447 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:11.692253 13273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:11.692478 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:11.692737 13273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:11.705456 13273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45913
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:11.706825 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:11.708489 13273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:11.754355 13308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.757236 13309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:11.761828 13312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:11.762390 13273 server_base.cc:1034] running on GCE node
I20260929 00:57:11.763383 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:11.771727 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:11.775149 13273 hybrid_clock.cc:648] HybridClock initialized: now 1790643431775092 us; error 94 us; skew 500 ppm
I20260929 00:57:11.775576 13273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:11 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:11.779258 13273 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:11.781364 13273 webserver.cc:460] Webserver started at http://127.0.97.1:45913/ using document root <none> and password file <none>
I20260929 00:57:11.782179 13273 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:11.782341 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:57:11 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:11.817391 13273 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260929 00:57:11.821659 13319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.823181 13273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:57:11.823377 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "0476223715af4950ab895cb170cb9521"
format_stamp: "Formatted at 2026-09-29 00:57:08 on dist-test-slave-pzsx"
server_key: "5d72940ba9d8ad86d60afdf00756d9e3"
server_key_iv: "e2e228e56ac97cf7ae58b81144587045"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:11.824491 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:11.855492 13273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:11.860780 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:11.861124 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:11.868538 13273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:11.871372 13273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:11.871634 13273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 00:57:11.871834 13273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:11.871966 13273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:11.972618 13273 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34793
I20260929 00:57:11.972698 13432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34793 every 8 connection(s)
I20260929 00:57:11.974589 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:11.983937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13273
I20260929 00:57:11.984750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44723
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44457
--webserver_port=33085
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:12 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:12.021535 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:11.978139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.1:54523)
0929 00:57:11.990256 (+ 12117us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:11.990323 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:11.992446 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:11.997474 (+ 5028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:11.997506 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:11.998158 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:11.999012 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:11.999042 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.002034 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.002043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.004299 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.004325 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.005960 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.005969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.006255 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.007543 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.007576 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.011500 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.016218 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.016227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.016245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.016898 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.017477 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.017482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.017486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.017735 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.018811 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.018820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.019222 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.019557 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.020114 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11795,"thread_start_us":11601,"threads_started":1}
I20260929 00:57:12.024895 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:11.978044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54523 (local address 127.0.97.62:44457)
0929 00:57:11.995623 (+ 17579us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:11.995638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:11.995680 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:57:11.996185 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:11.996212 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:11.996801 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:11.999304 (+ 2503us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:11.999764 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.001851 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.004578 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.005686 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.005898 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.011877 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.011890 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.011913 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.011968 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.016032 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.017104 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.017113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.017122 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.017336 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.017897 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.017901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.017905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.018649 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.022712 (+ 4063us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.022967 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.023368 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17327,"thread_start_us":18000,"threads_started":1}
I20260929 00:57:12.028477 13433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:12.028815 13433 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.029714 13433 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
W20260929 00:57:12.040490 13433 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44457 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 00:57:12.040817 13433 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.041817 13433 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:12.043372 13433 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.043706 13433 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
W20260929 00:57:12.044978 13433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:57:12.413048 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:12.413340 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:12.413532 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:12.421065 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:12.421203 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:12.421389 13437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:12.431514 13437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33085
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:12.433130 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:12.434398 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:12.449656 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:12.450138 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.450582 13437 server_base.cc:1034] running on GCE node
W20260929 00:57:12.450193 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:12.451303 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:12.452308 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:12.453522 13437 hybrid_clock.cc:648] HybridClock initialized: now 1790643432453461 us; error 53 us; skew 500 ppm
I20260929 00:57:12.453878 13437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:12 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:12.461722 13437 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:12.470115 13437 webserver.cc:460] Webserver started at http://127.0.97.2:33085/ using document root <none> and password file <none>
I20260929 00:57:12.470889 13437 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:12.471035 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:57:12 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:12.515194 13437 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.015s sys 0.000s
I20260929 00:57:12.518477 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.520234 13437 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 00:57:12.520430 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "0f79ee68802e4f7abb681c8864afa440"
format_stamp: "Formatted at 2026-09-29 00:57:09 on dist-test-slave-pzsx"
server_key: "37f3a6f86b35a1d91d9f98abb2a2b560"
server_key_iv: "1f541b2f22a4828e1f97c65ac178dcd6"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:12.521610 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:12.576788 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:12.581349 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:12.581730 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:12.588918 13437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:12.590806 13437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:12.590950 13437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.591082 13437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:12.591202 13437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:12.632925 13437 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44723
I20260929 00:57:12.633024 13576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44723 every 8 connection(s)
I20260929 00:57:12.634809 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:12.639374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13437
I20260929 00:57:12.640132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39557
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44457
--webserver_port=33075
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:12 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 00:57:12 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:12.670945 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.637742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59749 (local address 127.0.97.62:44457)
0929 00:57:12.638222 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:12.638229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:12.648747 (+ 10518us) server_negotiation.cc:407] Connection header received
0929 00:57:12.651563 (+ 2816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:12.651570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:12.651675 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:12.651769 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:12.654182 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.655372 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.657789 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.658448 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.658553 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.663516 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:12.663523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:12.663530 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:12.663573 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:12.667198 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.668117 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.668122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.668126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.668281 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:12.668838 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:12.668843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:12.668846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:12.669210 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:12.669359 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:12.670337 (+ 978us) server_negotiation.cc:299] Negotiation successful
0929 00:57:12.670562 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260929 00:57:12.673164 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:12.638527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.2:59749)
0929 00:57:12.648555 (+ 10028us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:12.648618 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:12.651135 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:12.652390 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:12.652402 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:12.653089 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:12.653969 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.653995 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.655601 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.655606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.657621 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:12.657636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:12.658906 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:12.658913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:12.659205 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:12.660059 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:12.660089 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:12.663252 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:12.667360 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.667370 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.667389 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.667979 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.668422 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:12.668426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:12.668429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:12.668711 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:12.669449 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:12.669457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:12.669879 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 00:57:12.671188 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0929 00:57:12.671721 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9626,"thread_start_us":13860,"threads_started":1}
I20260929 00:57:12.675034 13577 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:12.675328 13577 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:12.676102 13577 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:12.678141 13236 ts_manager.cc:194] Registered new tserver with Master: 0f79ee68802e4f7abb681c8864afa440 (127.0.97.2:44723)
I20260929 00:57:12.680194 13236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59749
I20260929 00:57:13.045130 13433 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.045558 13433 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:13.047648 13236 ts_manager.cc:194] Registered new tserver with Master: 0476223715af4950ab895cb170cb9521 (127.0.97.1:34793)
I20260929 00:57:13.048844 13236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54523
Sep 29 00:57:13 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 00:57:13.067916 13581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:13.068357 13581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:13.068565 13581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:13.079842 13581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:13.080077 13581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:13.080286 13581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:13.095427 13581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33075
--tserver_master_addrs=127.0.97.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:13.097334 13581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:13.099036 13581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:13.143795 13587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.144870 13588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:13.145562 13591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:13.153716 13581 server_base.cc:1034] running on GCE node
I20260929 00:57:13.154763 13581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:13.156149 13581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:13.157449 13581 hybrid_clock.cc:648] HybridClock initialized: now 1790643433157370 us; error 69 us; skew 500 ppm
I20260929 00:57:13.157852 13581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:13 dist-test-slave-pzsx krb5kdc[12225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:13.162380 13581 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:13.165977 13581 webserver.cc:460] Webserver started at http://127.0.97.3:33075/ using document root <none> and password file <none>
I20260929 00:57:13.166893 13581 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:13.167112 13581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 00:57:13 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:13.192833 13581 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:57:13.196542 13598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.198413 13581 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:57:13.198593 13581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "5b82665cca5f44c7a4bcc701a62308e1"
format_stamp: "Formatted at 2026-09-29 00:57:10 on dist-test-slave-pzsx"
server_key: "dfefa5ba5ab7babe60f212dd3b10a680"
server_key_iv: "23d652b3aa1a174b651f4bae1a85bbdc"
server_key_version: "kuduclusterkey@0"
I20260929 00:57:13.199913 13581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:13.228410 13581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:13.233566 13581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:13.233860 13581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:13.234979 13581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:13.236693 13581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:13.236804 13581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.236901 13581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:13.236974 13581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:13.298611 13581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39557
I20260929 00:57:13.298652 13710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39557 every 8 connection(s)
I20260929 00:57:13.300451 13581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:13.302340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13581
Sep 29 00:57:13 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:13.379948 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.305093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.0.97.62:44457)
0929 00:57:13.349781 (+ 44688us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.349786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.349815 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:57:13.349878 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.349884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.349968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.350066 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.352605 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.353691 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.358373 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.358948 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.359062 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.367100 (+ 8038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.367107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.367112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.367153 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.371137 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.372084 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.372090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.372094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.372233 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.373230 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.373237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.373243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.373584 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.373866 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.374825 (+ 959us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.375045 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44589,"thread_start_us":44423,"threads_started":1}
I20260929 00:57:13.383769 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.304861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60169 (local address 127.0.97.62:44457)
0929 00:57:13.348357 (+ 43496us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.348364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.356539 (+ 8175us) server_negotiation.cc:407] Connection header received
0929 00:57:13.360325 (+ 3786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.360330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.360419 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.360487 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.365232 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.366446 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.368940 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.369649 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.369752 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.375418 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.375429 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.375434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.375469 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.378954 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.380221 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.380226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.380230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.380360 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.381413 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.381418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.381421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.381813 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.381948 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.383211 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.383419 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43353,"thread_start_us":44385,"threads_started":1}
I20260929 00:57:13.384995 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.303930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44457 (local address 127.0.97.3:60169)
0929 00:57:13.353976 (+ 50046us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.354047 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.356397 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.361088 (+ 4691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.361100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.362611 (+ 1511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:13.363500 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.363527 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.366669 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.366674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.368752 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.368789 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.370128 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.370136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.370403 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.371621 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:13.371666 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:13.374517 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:13.379119 (+ 4602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.379129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.379148 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.379733 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.380555 (+ 822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:13.380560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:13.380564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:13.381303 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:13.382017 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:13.382025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:13.382711 (+ 686us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.383068 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.383606 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48355,"thread_start_us":55199,"threads_started":1}
I20260929 00:57:13.386816 13711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44457
I20260929 00:57:13.387115 13711 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:13.387955 13711 heartbeater.cc:511] Master 127.0.97.62:44457 requested a full tablet report, sending...
I20260929 00:57:13.389478 13236 ts_manager.cc:194] Registered new tserver with Master: 5b82665cca5f44c7a4bcc701a62308e1 (127.0.97.3:39557)
I20260929 00:57:13.390267 13236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60169
I20260929 00:57:13.395105 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 00:57:13 dist-test-slave-pzsx krb5kdc[12225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 00:57:13.428265 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.406814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.0.97.62:44457)
0929 00:57:13.407000 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.407007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.413129 (+ 6122us) server_negotiation.cc:407] Connection header received
0929 00:57:13.413352 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.413357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.413427 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.413534 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.415058 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.415770 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.417410 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.418005 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.418100 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.421747 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.421752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.421756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.421795 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.424856 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.425645 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.425650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.425654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.425784 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.426167 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.426173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.426176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.426562 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.426809 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.427741 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.427983 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:13.434458 13236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:13.445912 13236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:13.516160 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.478812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.97.2:44723)
0929 00:57:13.505386 (+ 26574us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.505395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.505425 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:57:13.505525 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.505534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.505826 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.506052 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.508715 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.510348 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.513178 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.514621 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.514753 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.514981 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.515635 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.515822 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26410,"thread_start_us":26302,"threads_started":1}
I20260929 00:57:13.517325 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.478640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44723 (local address 127.0.0.1:57632)
0929 00:57:13.501754 (+ 23114us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.501820 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.502066 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.506376 (+ 4310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.506385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.506423 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.506815 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.506829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.510536 (+ 3707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.510541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.512985 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.513004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.515136 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.515143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.515347 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.515401 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.516699 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.516918 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22695,"thread_start_us":25063,"threads_started":1}
I20260929 00:57:13.526305 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.505090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.97.3:39557)
0929 00:57:13.510752 (+ 5662us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.510761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.514248 (+ 3487us) server_negotiation.cc:407] Connection header received
0929 00:57:13.514450 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.514458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.516718 (+ 2260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.516964 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.517836 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.519636 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.523151 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.524663 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.524791 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.525025 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.525784 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.525979 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5478,"thread_start_us":5318,"threads_started":1}
I20260929 00:57:13.527009 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.504799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39557 (local address 127.0.0.1:44962)
0929 00:57:13.514143 (+ 9344us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.514162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.514304 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.516964 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.516970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.516996 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.517460 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.517476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.519835 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.519840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.521804 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.521819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.525185 (+ 3366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.525191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.525361 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.525396 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.525488 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.525613 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9222,"thread_start_us":9572,"threads_started":1}
I20260929 00:57:13.536020 13646 tablet_service.cc:1467] Processing CreateTablet for tablet 5b697eb321554decb57f359d71c449f4 (DEFAULT_TABLE table=test-table [id=4d823f62187b4fbe8e9850d60d687d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.537660 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.514717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48628)
0929 00:57:13.517749 (+ 3032us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.517768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.517884 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.529419 (+ 11535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.529425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.529454 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.529626 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.529637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.532122 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.532130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.533953 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.533968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.536938 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.536944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.537148 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.537185 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.537254 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.537392 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2938,"thread_start_us":17098,"threads_started":1}
I20260929 00:57:13.537698 13646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b697eb321554decb57f359d71c449f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.538681 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.515066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.97.1:34793)
0929 00:57:13.528712 (+ 13646us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.528721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.528750 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:57:13.528862 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.528870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.529127 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.529824 (+ 697us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.530067 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.531573 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.534148 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.536752 (+ 2604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.537912 (+ 1160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.538139 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.538210 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.538370 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13477,"thread_start_us":13304,"threads_started":1}
I20260929 00:57:13.544092 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 5b697eb321554decb57f359d71c449f4 (DEFAULT_TABLE table=test-table [id=4d823f62187b4fbe8e9850d60d687d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.544626 13368 tablet_service.cc:1467] Processing CreateTablet for tablet 5b697eb321554decb57f359d71c449f4 (DEFAULT_TABLE table=test-table [id=4d823f62187b4fbe8e9850d60d687d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:13.545828 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b697eb321554decb57f359d71c449f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.545857 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b697eb321554decb57f359d71c449f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:13.567272 13733 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Bootstrap starting.
I20260929 00:57:13.573104 13733 tablet_bootstrap.cc:654] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.573814 13734 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Bootstrap starting.
I20260929 00:57:13.574538 13733 log.cc:824] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.579811 13734 tablet_bootstrap.cc:654] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.582167 13735 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Bootstrap starting.
I20260929 00:57:13.584601 13733 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: No bootstrap required, opened a new log
I20260929 00:57:13.584957 13733 ts_tablet_manager.cc:1397] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260929 00:57:13.587406 13735 tablet_bootstrap.cc:654] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:13.588701 13735 log.cc:824] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.590456 13734 log.cc:824] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:13.598534 13733 raft_consensus.cc:348] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.599478 13733 raft_consensus.cc:374] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.599577 13735 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: No bootstrap required, opened a new log
I20260929 00:57:13.599666 13733 raft_consensus.cc:729] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f79ee68802e4f7abb681c8864afa440, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.599987 13735 ts_tablet_manager.cc:1397] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260929 00:57:13.602129 13734 tablet_bootstrap.cc:492] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: No bootstrap required, opened a new log
I20260929 00:57:13.602404 13734 ts_tablet_manager.cc:1397] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
I20260929 00:57:13.606340 13733 consensus_queue.cc:260] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.611049 13735 raft_consensus.cc:348] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.611572 13735 raft_consensus.cc:374] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.611738 13735 raft_consensus.cc:729] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0476223715af4950ab895cb170cb9521, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.612537 13735 consensus_queue.cc:260] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.615005 13577 heartbeater.cc:503] Master 127.0.97.62:44457 was elected leader, sending a full tablet report...
I20260929 00:57:13.614856 13734 raft_consensus.cc:348] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.615257 13734 raft_consensus.cc:374] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:13.615347 13734 raft_consensus.cc:729] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b82665cca5f44c7a4bcc701a62308e1, State: Initialized, Role: FOLLOWER
I20260929 00:57:13.616110 13733 ts_tablet_manager.cc:1428] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Time spent starting tablet: real 0.031s user 0.014s sys 0.001s
I20260929 00:57:13.616006 13734 consensus_queue.cc:260] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.628265 13433 heartbeater.cc:503] Master 127.0.97.62:44457 was elected leader, sending a full tablet report...
I20260929 00:57:13.629127 13735 ts_tablet_manager.cc:1428] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Time spent starting tablet: real 0.029s user 0.012s sys 0.013s
W20260929 00:57:13.643916 13578 tablet.cc:2369] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:13.649092 13734 ts_tablet_manager.cc:1428] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Time spent starting tablet: real 0.047s user 0.009s sys 0.012s
I20260929 00:57:13.649663 13711 heartbeater.cc:503] Master 127.0.97.62:44457 was elected leader, sending a full tablet report...
I20260929 00:57:13.710840 13741 raft_consensus.cc:482] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:13.711165 13741 raft_consensus.cc:504] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.713375 13741 leader_election.cc:288] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f79ee68802e4f7abb681c8864afa440 (127.0.97.2:44723), 0476223715af4950ab895cb170cb9521 (127.0.97.1:34793)
I20260929 00:57:13.720569 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.714238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41673 (local address 127.0.97.1:34793)
0929 00:57:13.714411 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.714417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.714486 (+ 69us) server_negotiation.cc:407] Connection header received
0929 00:57:13.714735 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.714740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.714825 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.714913 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.715591 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.716611 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.718377 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.719565 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.719703 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.719811 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.720165 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.720295 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:57:13.720777 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.713628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44723 (local address 127.0.97.3:47829)
0929 00:57:13.713784 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.713804 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.713934 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.714864 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.714868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.714890 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.715061 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.715069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.716322 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.716327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.718427 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.718442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.720016 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.720022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.720208 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.720240 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.720335 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.720461 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260929 00:57:13.720641 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.713979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.97.3:41673)
0929 00:57:13.714400 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.714416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:13.714568 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:13.715055 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:13.715059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:13.715082 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:13.715424 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.715434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.716760 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.716765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.718232 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:13.718243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.719765 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.719771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:13.719955 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.719989 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:57:13.720126 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 00:57:13.720257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":402,"threads_started":1}
I20260929 00:57:13.721081 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.714330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47829 (local address 127.0.97.2:44723)
0929 00:57:13.714573 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.714580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.714597 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:13.714673 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.714678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.714759 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.714851 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:13.715187 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.716167 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.718575 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.719828 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.719988 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.720079 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.720723 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.720840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 00:57:13.721993 13532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b697eb321554decb57f359d71c449f4" candidate_uuid: "5b82665cca5f44c7a4bcc701a62308e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f79ee68802e4f7abb681c8864afa440" is_pre_election: true
I20260929 00:57:13.721994 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b697eb321554decb57f359d71c449f4" candidate_uuid: "5b82665cca5f44c7a4bcc701a62308e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0476223715af4950ab895cb170cb9521" is_pre_election: true
I20260929 00:57:13.722471 13388 raft_consensus.cc:2454] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b82665cca5f44c7a4bcc701a62308e1 in term 0.
I20260929 00:57:13.722477 13532 raft_consensus.cc:2454] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b82665cca5f44c7a4bcc701a62308e1 in term 0.
I20260929 00:57:13.723080 13602 leader_election.cc:302] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f79ee68802e4f7abb681c8864afa440, 5b82665cca5f44c7a4bcc701a62308e1; no voters:
I20260929 00:57:13.723469 13741 raft_consensus.cc:2781] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:13.723616 13741 raft_consensus.cc:482] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:13.723727 13741 raft_consensus.cc:3037] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:13.726035 13741 raft_consensus.cc:504] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.726730 13741 leader_election.cc:288] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [CANDIDATE]: Term 1 election: Requested vote from peers 0f79ee68802e4f7abb681c8864afa440 (127.0.97.2:44723), 0476223715af4950ab895cb170cb9521 (127.0.97.1:34793)
I20260929 00:57:13.727226 13532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b697eb321554decb57f359d71c449f4" candidate_uuid: "5b82665cca5f44c7a4bcc701a62308e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f79ee68802e4f7abb681c8864afa440"
I20260929 00:57:13.727344 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b697eb321554decb57f359d71c449f4" candidate_uuid: "5b82665cca5f44c7a4bcc701a62308e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0476223715af4950ab895cb170cb9521"
I20260929 00:57:13.727428 13532 raft_consensus.cc:3037] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:13.727584 13388 raft_consensus.cc:3037] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 0 FOLLOWER]: Advancing to term 1
W20260929 00:57:13.727860 13434 tablet.cc:2369] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:13.730170 13388 raft_consensus.cc:2454] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b82665cca5f44c7a4bcc701a62308e1 in term 1.
I20260929 00:57:13.730545 13602 leader_election.cc:302] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0476223715af4950ab895cb170cb9521, 5b82665cca5f44c7a4bcc701a62308e1; no voters:
I20260929 00:57:13.730620 13532 raft_consensus.cc:2454] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b82665cca5f44c7a4bcc701a62308e1 in term 1.
I20260929 00:57:13.730870 13741 raft_consensus.cc:2781] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:13.731344 13741 raft_consensus.cc:686] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5b82665cca5f44c7a4bcc701a62308e1, State: Running, Role: LEADER
I20260929 00:57:13.731812 13741 consensus_queue.cc:237] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } }
I20260929 00:57:13.736362 13236 catalog_manager.cc:5495] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b82665cca5f44c7a4bcc701a62308e1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5b82665cca5f44c7a4bcc701a62308e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b82665cca5f44c7a4bcc701a62308e1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:13.778129 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.773058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.0.97.3:39557)
0929 00:57:13.773247 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.773252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.773268 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:13.773355 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.773358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.773444 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.773561 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:13.773838 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.774545 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.775958 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.776376 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.776471 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.777105 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:13.777597 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:13.777707 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.777775 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.777836 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:57:13.786092 13388 raft_consensus.cc:1261] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 1 FOLLOWER]: Refusing update from remote peer 5b82665cca5f44c7a4bcc701a62308e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:13.786139 13532 raft_consensus.cc:1261] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 1 FOLLOWER]: Refusing update from remote peer 5b82665cca5f44c7a4bcc701a62308e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:13.787017 13750 consensus_queue.cc:1035] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0476223715af4950ab895cb170cb9521" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:13.787357 13741 consensus_queue.cc:1035] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f79ee68802e4f7abb681c8864afa440" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:13.802641 13755 mvcc.cc:205] Tried to move back new op lower bound from 7334475504778412032 to 7334475504569729024. Current Snapshot: MvccSnapshot[applied={T|T < 7334475504778412032}]
I20260929 00:57:13.811307 13236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:13.811561 13236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:13.814862 13236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e016cfc83a4342b0d7517f1e78ff40: Sending DeleteTablet for 3 replicas of tablet 5b697eb321554decb57f359d71c449f4
I20260929 00:57:13.815882 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b697eb321554decb57f359d71c449f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260929 00:57:13.816075 13646 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b697eb321554decb57f359d71c449f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44962
I20260929 00:57:13.816116 13368 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b697eb321554decb57f359d71c449f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48628
I20260929 00:57:13.816922 13762 tablet_replica.cc:330] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: stopping tablet replica
I20260929 00:57:13.817487 13762 raft_consensus.cc:2229] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:13.818110 13762 raft_consensus.cc:2258] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:13.818768 13763 tablet_replica.cc:330] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: stopping tablet replica
I20260929 00:57:13.819314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13273
I20260929 00:57:13.824483 13763 raft_consensus.cc:2229] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:13.825146 13763 raft_consensus.cc:2258] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:13.826143 13762 ts_tablet_manager.cc:1905] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:13.829641 13764 tablet_replica.cc:330] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: stopping tablet replica
I20260929 00:57:13.830086 13764 raft_consensus.cc:2229] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:13.830204 13763 ts_tablet_manager.cc:1905] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:13.830655 13764 raft_consensus.cc:2258] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:13.833343 13764 ts_tablet_manager.cc:1905] T 5b697eb321554decb57f359d71c449f4 P 0476223715af4950ab895cb170cb9521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:13.835757 13762 ts_tablet_manager.cc:1918] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:13.836020 13762 log.cc:1196] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/5b697eb321554decb57f359d71c449f4
I20260929 00:57:13.836684 13762 ts_tablet_manager.cc:1939] T 5b697eb321554decb57f359d71c449f4 P 0f79ee68802e4f7abb681c8864afa440: Deleting consensus metadata
I20260929 00:57:13.838783 13223 catalog_manager.cc:4841] TS 0f79ee68802e4f7abb681c8864afa440 (127.0.97.2:44723): tablet 5b697eb321554decb57f359d71c449f4 (table test-table [id=4d823f62187b4fbe8e9850d60d687d8a]) successfully deleted
I20260929 00:57:13.840598 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.820890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.0.97.1:34793)
0929 00:57:13.821051 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.821057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.821273 (+ 216us) server_negotiation.cc:407] Connection header received
0929 00:57:13.821510 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.821515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.821600 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.821693 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.824068 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.825197 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.827629 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.828556 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.828680 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.832059 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.832071 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.832087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.832124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.835762 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.836717 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.836724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.836728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.836943 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.837392 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.837398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.837401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.837844 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.838157 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.839947 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.840187 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:57:13.841337 13428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:13.843653 13763 ts_tablet_manager.cc:1918] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:13.843844 13763 log.cc:1196] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/5b697eb321554decb57f359d71c449f4
I20260929 00:57:13.844846 13763 ts_tablet_manager.cc:1939] T 5b697eb321554decb57f359d71c449f4 P 5b82665cca5f44c7a4bcc701a62308e1: Deleting consensus metadata
I20260929 00:57:13.846743 13223 catalog_manager.cc:4841] TS 5b82665cca5f44c7a4bcc701a62308e1 (127.0.97.3:39557): tablet 5b697eb321554decb57f359d71c449f4 (table test-table [id=4d823f62187b4fbe8e9850d60d687d8a]) successfully deleted
I20260929 00:57:13.930262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13273
I20260929 00:57:13.947592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13437
W20260929 00:57:13.947571 13223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34793 (error 108)
I20260929 00:57:13.949939 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.949415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48650)
0929 00:57:13.949617 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:13.949746 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 00:57:13.950215 13223 catalog_manager.cc:4584] TS 0476223715af4950ab895cb170cb9521 (127.0.97.1:34793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b697eb321554decb57f359d71c449f4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
I20260929 00:57:13.963119 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:13.948557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.2:44723)
0929 00:57:13.948744 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:13.948752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:13.948774 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:57:13.948961 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:13.948966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:13.949079 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:13.949193 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:13.950787 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.951919 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.953238 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:13.953867 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:13.954004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:13.956422 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:13.956439 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:13.956458 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:13.956535 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:13.960211 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.961032 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.961042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.961046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.961252 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:13.961591 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:13.961596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:13.961600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:13.962044 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:13.962332 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:13.962516 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 00:57:13.962752 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:57:13.963874 13572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:14.003863 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.003325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48658)
0929 00:57:14.003575 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.003696 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 00:57:14.005137 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.004678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48670)
0929 00:57:14.004846 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.004940 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 00:57:14.057085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13437
I20260929 00:57:14.066008 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.065419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48674)
0929 00:57:14.065681 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.065815 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 00:57:14.067348 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.066825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48680)
0929 00:57:14.067033 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.067162 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:57:14.074371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13581
I20260929 00:57:14.090306 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.075339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.97.3:39557)
0929 00:57:14.075519 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:14.075523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:14.075539 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:57:14.075592 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:14.075596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:14.075652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:14.075745 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:14.077644 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.078496 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.080467 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:14.080959 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:14.081059 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:14.084144 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:14.084155 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:14.084169 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:14.084204 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:14.087363 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.088172 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.088179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.088181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.088324 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:14.088793 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:14.088797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:14.088799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:14.089138 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:14.089408 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:14.089792 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 00:57:14.089964 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:57:14.091149 13706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:14.147883 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.147328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48692)
0929 00:57:14.147590 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.147705 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 00:57:14.149107 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:14.148619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34793 (local address 127.0.0.1:48694)
0929 00:57:14.148838 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:14.148936 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 00:57:14.211498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13581
I20260929 00:57:14.230367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13203
I20260929 00:57:14.231549 13266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:14.343221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13203
I20260929 00:57:14.364385 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 00:57:14.700369 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 00:57:14.700634 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 00:57:15.046277 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 00:57:15.046 UTC [12255] LOG: received smart shutdown request
2026-09-29 00:57:15.049 UTC [12255] LOG: background worker "logical replication launcher" (PID 12264) exited with exit code 1
2026-09-29 00:57:15.049 UTC [12259] LOG: shutting down
2026-09-29 00:57:15.210 UTC [12255] LOG: database system is shut down
2026-09-29T00:57:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50690 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:15 dist-test-slave-pzsx krb5kdc[13778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:15 dist-test-slave-pzsx krb5kdc[13778](info): set up 2 sockets
Sep 29 00:57:15 dist-test-slave-pzsx krb5kdc[13778](info): commencing operation
krb5kdc: starting...
W20260929 00:57:16.424938 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:57:16 dist-test-slave-pzsx krb5kdc[13778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:57:16.648290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43803
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:16.973340 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:16.973613 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:16.973712 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:16.980818 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:16.980918 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:16.980970 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:16.981025 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:16.981069 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:16.990818 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:16.992074 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:16.993433 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:17.005896 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.005851 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.005916 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:17.006824 13791 server_base.cc:1034] running on GCE node
I20260929 00:57:17.007611 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:17.009114 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:17.010336 13791 hybrid_clock.cc:648] HybridClock initialized: now 1790643437010294 us; error 49 us; skew 500 ppm
I20260929 00:57:17.010689 13791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:17.013871 13791 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:17.015270 13791 webserver.cc:460] Webserver started at http://127.0.97.62:41371/ using document root <none> and password file <none>
I20260929 00:57:17.015951 13791 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:17.016101 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:17.016403 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:17.019368 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "71ebe1a487514f6b9aec441f39d8ee46"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "56f4d5780373adc1f73026a7902435ad"
server_key_iv: "b1cb4d6a40e5b3e9e00337af1deff6a2"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.020153 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "71ebe1a487514f6b9aec441f39d8ee46"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "56f4d5780373adc1f73026a7902435ad"
server_key_iv: "b1cb4d6a40e5b3e9e00337af1deff6a2"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.025177 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:57:17.029402 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:17.030679 13791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:17.030869 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "71ebe1a487514f6b9aec441f39d8ee46"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "56f4d5780373adc1f73026a7902435ad"
server_key_iv: "b1cb4d6a40e5b3e9e00337af1deff6a2"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.031067 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:17.046255 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:17.050221 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:17.050505 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:17.068775 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43803
I20260929 00:57:17.068805 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43803 every 8 connection(s)
I20260929 00:57:17.070251 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:17.073915 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:17.080472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13791
I20260929 00:57:17.080727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:57:17.081398 388 external_mini_cluster.cc:1428] Setting key 7cdeff52295987ebdd1a0c8dba0e1f87
I20260929 00:57:17.080821 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260929 00:57:17.106240 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.083995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43803 (local address 127.0.0.1:52464)
0929 00:57:17.084980 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.085044 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.087890 (+ 2846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.088767 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.088774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.089394 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:17.089748 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.089776 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.091298 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.091306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.093280 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.093297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.094192 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.094201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.094436 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.095163 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:17.095195 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:17.097859 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:17.102226 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.102235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.102244 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.102639 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.103116 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.103120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.103122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.103258 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.104220 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:17.104225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:17.104434 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.104717 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.105166 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":442,"threads_started":1}
I20260929 00:57:17.106285 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.083995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.97.62:43803)
0929 00:57:17.085964 (+ 1969us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.085970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.086012 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:57:17.088246 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.088251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.088473 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.088746 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:17.089966 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.091129 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.093413 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.093954 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.094081 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.098104 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:17.098116 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:17.098134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:17.098176 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:17.102086 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.102788 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.102797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.102801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.102999 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.103390 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.103397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.103402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.104089 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:17.104341 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.104915 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.105313 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":1786,"spinlock_wait_cycles":57728,"thread_start_us":1574,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 00:57:17.109282 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:57:17.110603 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.110077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39038)
0929 00:57:17.110301 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.110442 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 00:57:17.110908 13809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
I20260929 00:57:17.112651 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.112176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39048)
0929 00:57:17.112358 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.112467 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 00:57:17.112874 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 00:57:17.133579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37315
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:17.162236 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } attempt: 1
I20260929 00:57:17.163517 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.162982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39054)
0929 00:57:17.163220 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.163330 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 00:57:17.164572 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.164148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39068)
0929 00:57:17.164290 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.164369 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 00:57:17.164764 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
I20260929 00:57:17.233044 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } attempt: 2
I20260929 00:57:17.234345 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.233791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39078)
0929 00:57:17.234021 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.234134 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 00:57:17.235427 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.235017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39088)
0929 00:57:17.235173 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.235274 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:57:17.235643 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
I20260929 00:57:17.326963 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } attempt: 3
I20260929 00:57:17.328356 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.327760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39096)
0929 00:57:17.328027 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.328161 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 00:57:17.329450 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.329058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39112)
0929 00:57:17.329206 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.329300 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:57:17.329640 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
W20260929 00:57:17.465339 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:17.465651 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:17.465809 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:57:17.472865 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } attempt: 4
I20260929 00:57:17.474174 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.473584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39128)
0929 00:57:17.473880 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.474013 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260929 00:57:17.474210 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:17.474311 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:17.474404 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:17.474539 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:17.474622 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:17.475229 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.474840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39132)
0929 00:57:17.474992 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.475085 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 00:57:17.475428 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37315: connect: Connection refused (error 111)
I20260929 00:57:17.485270 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:17.486531 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:17.487762 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:17.499559 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.499500 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.499581 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:17.500597 13865 server_base.cc:1034] running on GCE node
I20260929 00:57:17.501479 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:17.502974 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:17.504185 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790643437504128 us; error 57 us; skew 500 ppm
I20260929 00:57:17.504596 13865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:17.510821 13865 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 00:57:17.512336 13865 webserver.cc:460] Webserver started at http://127.0.97.61:34547/ using document root <none> and password file <none>
I20260929 00:57:17.513120 13865 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:17.513268 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:17.513574 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:17.516753 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data/instance:
uuid: "4c81e145eb714b6daa5a01c25a02b6c0"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "da8520b2200e953bf21f52fb0b5d9e4a"
server_key_iv: "db62e3ce63b947268eceba6eb1abe305"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.517592 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal/instance:
uuid: "4c81e145eb714b6daa5a01c25a02b6c0"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "da8520b2200e953bf21f52fb0b5d9e4a"
server_key_iv: "db62e3ce63b947268eceba6eb1abe305"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.522861 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:57:17.526391 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:17.527683 13865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:17.527871 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "4c81e145eb714b6daa5a01c25a02b6c0"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "da8520b2200e953bf21f52fb0b5d9e4a"
server_key_iv: "db62e3ce63b947268eceba6eb1abe305"
server_key_version: "encryptionkey@0"
I20260929 00:57:17.528098 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:17.550534 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:17.554525 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:17.554844 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:17.573256 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37315
I20260929 00:57:17.573285 13931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37315 every 8 connection(s)
I20260929 00:57:17.574697 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
I20260929 00:57:17.575388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13865
I20260929 00:57:17.575598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-1/wal/instance
I20260929 00:57:17.576155 388 external_mini_cluster.cc:1428] Setting key f0af0a980a24bf11d83578d12177b460
I20260929 00:57:17.578679 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:17.586813 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:17.608474 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.590304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.97.62:43803)
0929 00:57:17.590552 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.590561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.591784 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 00:57:17.593724 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.593731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.593834 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.593964 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:17.595403 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.596568 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.598115 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.598595 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.598694 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.602657 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:17.602662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:17.602665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:17.602697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:17.605383 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.606144 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.606151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.606154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.606278 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.606759 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.606765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.606768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.607046 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:17.607199 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.607927 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.608156 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 00:57:17.609577 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.590750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43803 (local address 127.0.0.1:52478)
0929 00:57:17.591599 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.591644 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.593437 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.594150 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.594158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.594664 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:17.595217 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.595234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.596718 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.596722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.597985 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.597997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.598791 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.598796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.598982 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.599693 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:17.599725 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:17.602429 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:17.605570 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.605580 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.605596 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.606026 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.606392 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.606397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.606400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.606639 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.607173 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:17.607181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:17.607563 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.607839 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.608352 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":390,"threads_started":1}
I20260929 00:57:17.611344 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 00:57:17.628165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33377
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:17.629465 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.612354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.97.61:37315)
0929 00:57:17.613082 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.613087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.613112 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:57:17.613153 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.613159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.613367 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.613477 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:17.614415 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.615431 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.617382 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.617931 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.618032 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.623528 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:17.623540 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:17.623555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:17.623587 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:17.626590 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.627272 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.627279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.627281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.627397 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.627834 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.627837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.627839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.628181 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:17.628422 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.628860 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.629133 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":578,"thread_start_us":443,"threads_started":1}
I20260929 00:57:17.629558 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.612072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39142)
0929 00:57:17.612310 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.612334 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.612439 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.613546 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.613553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.614051 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:17.614259 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.614272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.615593 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.615601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.617230 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.617248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.618086 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.618096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.618302 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.618932 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:17.618955 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:17.623323 (+ 4368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:17.626715 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.626719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.626722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.627124 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.627521 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.627527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.627530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.627708 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.628320 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:17.628327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:17.628549 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.628696 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.629035 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:57:17.632179 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:57:17.633514 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.632914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51062)
0929 00:57:17.633128 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.633308 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 00:57:17.633780 13883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.635540 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.635101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51078)
0929 00:57:17.635281 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.635398 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:57:17.635785 13932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.685128 13932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 1
I20260929 00:57:17.686349 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.685900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51082)
0929 00:57:17.686100 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.686200 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 00:57:17.687275 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.686947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51090)
0929 00:57:17.687113 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.687174 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:57:17.687522 13932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.755748 13932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 2
I20260929 00:57:17.756982 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.756546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51106)
0929 00:57:17.756741 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.756828 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 00:57:17.757899 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.757589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51114)
0929 00:57:17.757728 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.757801 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:57:17.758129 13932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.774739 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } attempt: 5
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:57:17.789089 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.775515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.0.97.61:37315)
0929 00:57:17.775731 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:17.775740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:17.775768 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:57:17.775884 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:17.775891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:17.776000 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:17.776140 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:17.776940 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.778045 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.779176 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.779622 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.779750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.783682 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:17.783686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:17.783690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:17.783720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:17.786365 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.787304 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.787308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.787310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.787420 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:17.787835 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:17.787838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:17.787840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:17.788099 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:17.788202 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:17.788701 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 00:57:17.788865 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 00:57:17.789446 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.775410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39154)
0929 00:57:17.775596 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.775618 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:17.775737 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:17.776158 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:17.776161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:17.776599 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:17.776792 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.776801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.778189 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.778192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.779031 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:17.779041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:17.779781 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:17.779788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:17.779968 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:17.780670 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:17.780692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:17.783489 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:17.786523 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.786530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.786536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.787180 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.787554 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:17.787559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:17.787562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:17.787728 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:17.788218 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:17.788222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:17.788407 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 00:57:17.788620 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 00:57:17.788952 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 00:57:17.790452 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:57:17.791380 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.791044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51118)
0929 00:57:17.791197 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.791265 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 00:57:17.792250 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.791945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51134)
0929 00:57:17.792086 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.792159 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 00:57:17.792429 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.843822 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 1
I20260929 00:57:17.845118 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.844664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51146)
0929 00:57:17.844855 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.844948 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 00:57:17.846079 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.845775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51154)
0929 00:57:17.845921 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.845991 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 00:57:17.846278 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.849314 13932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 3
I20260929 00:57:17.850291 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.849900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51156)
0929 00:57:17.850085 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.850159 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:57:17.851243 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.850867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51160)
0929 00:57:17.851086 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.851150 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 00:57:17.851428 13932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.914570 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 2
I20260929 00:57:17.915760 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.915304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51166)
0929 00:57:17.915511 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.915611 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 00:57:17.916782 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.916439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51182)
0929 00:57:17.916624 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.916688 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:57:17.916970 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
W20260929 00:57:17.959131 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:17.959517 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:17.959685 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:17.967007 13939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:17.967161 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:17.967267 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:17.967365 13939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:17.967443 13939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:17.977362 13939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43803,127.0.97.61:37315,127.0.97.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:17.978766 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:17.980106 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:17.991984 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.992040 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:17.992126 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:17.993155 13939 server_base.cc:1034] running on GCE node
I20260929 00:57:17.993960 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:17.994623 13932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 4
I20260929 00:57:17.995505 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:17.995692 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.995183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51188)
0929 00:57:17.995406 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.995487 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 00:57:17.996675 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:17.996344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51190)
0929 00:57:17.996487 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:17.996576 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 00:57:17.996717 13939 hybrid_clock.cc:648] HybridClock initialized: now 1790643437996655 us; error 54 us; skew 500 ppm
W20260929 00:57:17.996876 13932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:17.997138 13939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:17 dist-test-slave-pzsx krb5kdc[13778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:18.000306 13939 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 00:57:18.002067 13939 webserver.cc:460] Webserver started at http://127.0.97.60:40849/ using document root <none> and password file <none>
I20260929 00:57:18.002772 13939 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:18.002902 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:18.003252 13939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:18.006247 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data/instance:
uuid: "0c38b6684f0e43afa3611e78b368d8be"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "3dceb979b8ad28fb010793bb108381bb"
server_key_iv: "14f0ea313431c6764f2794f4d9e0e1b4"
server_key_version: "encryptionkey@0"
I20260929 00:57:18.007054 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal/instance:
uuid: "0c38b6684f0e43afa3611e78b368d8be"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "3dceb979b8ad28fb010793bb108381bb"
server_key_iv: "14f0ea313431c6764f2794f4d9e0e1b4"
server_key_version: "encryptionkey@0"
I20260929 00:57:18.012123 13939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 00:57:18.015537 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:18.016842 13939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:57:18.017032 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "0c38b6684f0e43afa3611e78b368d8be"
format_stamp: "Formatted at 2026-09-29 00:57:17 on dist-test-slave-pzsx"
server_key: "3dceb979b8ad28fb010793bb108381bb"
server_key_iv: "14f0ea313431c6764f2794f4d9e0e1b4"
server_key_version: "encryptionkey@0"
I20260929 00:57:18.017246 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:18.023161 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 3
I20260929 00:57:18.024200 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.023769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51196)
0929 00:57:18.023981 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.024062 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:57:18.025139 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.024822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51204)
0929 00:57:18.024986 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.025047 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:57:18.025321 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33377: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33377: connect: Connection refused (error 111)
I20260929 00:57:18.043916 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:18.048270 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:18.048617 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:18.066707 13939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33377
I20260929 00:57:18.066732 14005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33377 every 8 connection(s)
I20260929 00:57:18.068097 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
I20260929 00:57:18.071830 14006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:18.071875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 13939
I20260929 00:57:18.072055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643237839665-388-0/minicluster-data/master-2/wal/instance
I20260929 00:57:18.072570 388 external_mini_cluster.cc:1428] Setting key 17e49353928702d12b2db9913aa9ab91
I20260929 00:57:18.084401 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:18.100872 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.074394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.0.97.60:33377)
0929 00:57:18.079604 (+ 5210us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.079616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.079659 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:57:18.081484 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.081517 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.082055 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.082510 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.083861 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.085921 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.087673 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.088370 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.088685 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.093129 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.093139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.093152 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.093201 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.096835 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.097468 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.097476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.097484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.097634 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.097997 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.098002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.098005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.098450 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.098729 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.099147 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.099524 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4949,"thread_start_us":4659,"threads_started":1}
I20260929 00:57:18.102356 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.087330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.0.97.62:43803)
0929 00:57:18.087513 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.087517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.088431 (+ 914us) server_negotiation.cc:407] Connection header received
0929 00:57:18.088609 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.088614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.088694 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.088797 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.090209 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.090866 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.092467 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.092960 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.093100 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.096893 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.096897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.096900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.096932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.099621 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.100377 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.100381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.100383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.100573 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.100991 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.100997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.101001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.101282 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.101400 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.101855 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.102010 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:57:18.102430 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.087290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43803 (local address 127.0.0.1:52484)
0929 00:57:18.088271 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.088315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:18.088473 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:18.088878 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:18.088885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:18.089718 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:18.090055 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.090069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.091013 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.091018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.092325 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.092339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.093142 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.093146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.093286 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.093904 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:18.093919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:18.096695 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:18.099739 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.099750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.099759 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.100254 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.100675 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.100680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.100683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.100856 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.101409 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:18.101419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:18.101616 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 00:57:18.101782 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 00:57:18.102068 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":589,"threads_started":1}
I20260929 00:57:18.103943 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:18.117741 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.104709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37315 (local address 127.0.0.1:39160)
0929 00:57:18.104875 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.104892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:18.104987 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:18.105302 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:18.105307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:18.105750 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:18.105929 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.105943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.107165 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.107168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.108472 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.108485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.109210 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.109214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.109367 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.109926 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:18.109942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:18.112141 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:18.115379 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.115385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.115390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.115867 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.116215 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.116222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.116228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.116404 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.116885 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:18.116892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:18.117051 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 00:57:18.117207 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 00:57:18.117464 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 00:57:18.118067 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.104693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.0.97.61:37315)
0929 00:57:18.104860 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.104865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.104981 (+ 116us) server_negotiation.cc:407] Connection header received
0929 00:57:18.105099 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.105104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.105170 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.105238 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.106081 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.107027 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.108621 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.109073 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.109190 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.112338 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.112347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.112352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.112400 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.115261 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.115979 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.115982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.115985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.116110 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.116562 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.116565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.116567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.116791 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.116901 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.117276 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.117521 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:57:18.119174 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:57:18.119560 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.104394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.97.62:43803)
0929 00:57:18.104717 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.104722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.104739 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:18.104783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.104788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.104852 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.104972 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.106395 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.107332 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.109035 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.109548 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.109686 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.113816 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.113821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.113824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.113851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.116680 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.117421 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.117426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.117429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.117598 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.118030 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.118035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.118038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.118403 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.118661 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.119037 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.119260 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:57:18.132866 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.119915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51222)
0929 00:57:18.120076 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.120091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:18.120183 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:18.120478 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:18.120482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:18.120974 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:18.121138 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.121146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.122377 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.122381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.123319 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.123332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.123979 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.123987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.124150 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.124707 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:18.124723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:18.127041 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:18.130272 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.130278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.130282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.130713 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.131084 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.131087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.131089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.131261 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.131874 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:18.131878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:18.132000 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 00:57:18.132148 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 00:57:18.132569 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260929 00:57:18.133133 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.120082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.0.97.60:33377)
0929 00:57:18.120223 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.120229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.120246 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:57:18.120302 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.120307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.120372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.120471 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.121257 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.122263 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.123451 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.123825 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.123907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.127211 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.127218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.127224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.127259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.130141 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.130819 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.130824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.130828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.130975 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.131376 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.131380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.131381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.131748 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.131889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.132264 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.132731 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":223,"server-negotiator.queue_time_us":50}
I20260929 00:57:18.136622 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.121362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.0.97.61:37315)
0929 00:57:18.121602 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.121606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.121621 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:57:18.121711 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.121714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.121762 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.121831 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.123065 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.123902 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.125162 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.125851 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.125939 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.130593 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.130597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.130600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.130628 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.133792 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.134591 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.134596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.134599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.134747 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.135123 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.135130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.135136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.135473 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.135736 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.136033 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.136248 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260929 00:57:18.138943 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: Bootstrap starting.
I20260929 00:57:18.141251 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:18.142513 14006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:18.143821 14006 log.cc:824] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:18.146574 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: No bootstrap required, opened a new log
I20260929 00:57:18.160816 14006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.161506 14006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:18.161685 14006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c38b6684f0e43afa3611e78b368d8be, State: Initialized, Role: FOLLOWER
I20260929 00:57:18.162528 14006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.166703 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.147381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.0.97.62:43803)
0929 00:57:18.147588 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.147594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.147735 (+ 141us) server_negotiation.cc:407] Connection header received
0929 00:57:18.148041 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.148051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.148132 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.148212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.150584 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.151699 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.153961 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.154620 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.154709 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.158761 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.158769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.158774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.158813 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.162878 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.163732 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.163741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.163745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.163885 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.164357 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.164364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.164368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.164754 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.165044 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.166069 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.166305 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:18.166703 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.147057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.0.97.61:37315)
0929 00:57:18.147232 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.147238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.148220 (+ 982us) server_negotiation.cc:407] Connection header received
0929 00:57:18.148458 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.148464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.148589 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.148667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.151202 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.152232 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.154542 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.155183 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.155282 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.158786 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.158792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.158796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.158833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.163477 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.164390 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.164394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.164398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.164574 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.165030 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.165034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.165037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.165393 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.165638 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.166135 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.166355 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:57:18.169283 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.169647 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.170897 14006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:18.177577 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.148756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.0.97.60:33377)
0929 00:57:18.148980 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.148988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.159687 (+ 10699us) server_negotiation.cc:407] Connection header received
0929 00:57:18.159848 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.159856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.160185 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.160303 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.161903 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.162980 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.165777 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.166740 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.166865 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.170768 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.170772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.170775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.170805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.174657 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.175364 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.175371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.175373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.175525 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.175921 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.175925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.175927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.176259 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.176573 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.176928 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.177169 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 00:57:18.182283 14030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:18.182478 14030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:18.178742 13939 master_runner.cc:186] Error getting master registration for 127.0.97.62:43803: OK, instance_id { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" instance_seqno: 1790643437063386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:57:18.187911 13939 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:57:18.202517 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 4
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:57:18.217022 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.203138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.0.97.60:33377)
0929 00:57:18.203296 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.203302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.203336 (+ 34us) server_negotiation.cc:407] Connection header received
0929 00:57:18.203522 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.203527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.203633 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.203744 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.204474 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.205282 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.206601 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.207284 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.207399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.211011 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.211015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.211018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.211046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.214242 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.215034 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.215038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.215040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.215182 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.215683 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.215686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.215688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.215937 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.216041 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.216485 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.216707 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:57:18.217123 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.203064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51230)
0929 00:57:18.203264 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.203284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:18.203388 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:18.203761 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:18.203765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:18.204170 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:18.204344 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.204352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.205406 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.205412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.206437 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.206449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.207460 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.207468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.207704 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.208328 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:18.208353 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:18.210839 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:18.214374 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.214379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.214386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.214909 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.215323 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.215328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.215332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.215562 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.216054 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:18.216061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:18.216237 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 00:57:18.216400 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 00:57:18.216724 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:57:18.221374 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: Bootstrap starting.
I20260929 00:57:18.225083 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:18.226089 13858 log.cc:824] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:18.228766 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: No bootstrap required, opened a new log
I20260929 00:57:18.236748 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.237035 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:18.237185 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ebe1a487514f6b9aec441f39d8ee46, State: Initialized, Role: FOLLOWER
I20260929 00:57:18.237710 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.238616 14033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.239019 14033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.239748 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:57:18.250074 14044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:18.250295 14044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:18.251525 13791 master_runner.cc:186] Error getting master registration for 127.0.97.61:37315: OK, instance_id { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" instance_seqno: 1790643437567819 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:57:18.264714 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:57:18.296119 13932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } attempt: 5
W20260929 00:57:18.300688 13960 tablet.cc:2369] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:18.303356 13812 tablet.cc:2369] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 00:57:18 dist-test-slave-pzsx krb5kdc[13778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:57:18.315341 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.296778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33377 (local address 127.0.0.1:51242)
0929 00:57:18.297053 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:18.297075 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:18.297183 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:18.297570 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:18.297575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:18.298085 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:18.298291 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.298303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.299741 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.299747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.301225 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:18.301240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.302297 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.302303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:18.302522 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.303231 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:18.303255 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:18.306183 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:18.312414 (+ 6231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.312420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.312425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.313040 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.313456 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:18.313461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:18.313465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:18.313637 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:18.314299 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:18.314304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:18.314472 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 00:57:18.314622 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 00:57:18.314929 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 00:57:18.315413 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:18.296874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.0.97.60:33377)
0929 00:57:18.297089 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:18.297095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:18.297123 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:57:18.297311 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:18.297317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:18.297393 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:18.297508 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:18.298451 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.299578 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.301407 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:18.302132 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:18.302259 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:18.306387 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:18.306394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:18.306399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:18.306442 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:18.312272 (+ 5830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.313165 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.313172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.313176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.313329 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:18.313771 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:18.313781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:18.313784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:18.314177 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:18.314330 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:18.314768 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 00:57:18.315008 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 00:57:18.321821 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: Bootstrap starting.
I20260929 00:57:18.328253 13932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:18.329667 13932 log.cc:824] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:18.333568 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: No bootstrap required, opened a new log
I20260929 00:57:18.344641 13932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.345093 13932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:18.345218 13932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c81e145eb714b6daa5a01c25a02b6c0, State: Initialized, Role: FOLLOWER
I20260929 00:57:18.345887 13932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.349406 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.349901 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.350819 13932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:57:18.369596 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:57:18.369835 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:57:18.370627 13865 master_runner.cc:263] No leader master found from master 4c81e145eb714b6daa5a01c25a02b6c0
I20260929 00:57:18.370765 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:57:18.395840 14033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:18.396139 14033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.397775 14033 leader_election.cc:288] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c81e145eb714b6daa5a01c25a02b6c0 (127.0.97.61:37315), 0c38b6684f0e43afa3611e78b368d8be (127.0.97.60:33377)
I20260929 00:57:18.398845 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71ebe1a487514f6b9aec441f39d8ee46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c38b6684f0e43afa3611e78b368d8be" is_pre_election: true
I20260929 00:57:18.399451 13981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ebe1a487514f6b9aec441f39d8ee46 in term 0.
I20260929 00:57:18.400153 13809 leader_election.cc:302] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c38b6684f0e43afa3611e78b368d8be, 71ebe1a487514f6b9aec441f39d8ee46; no voters:
I20260929 00:57:18.401245 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71ebe1a487514f6b9aec441f39d8ee46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" is_pre_election: true
I20260929 00:57:18.401803 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ebe1a487514f6b9aec441f39d8ee46 in term 0.
I20260929 00:57:18.402740 14033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:18.402884 14033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:18.402985 14033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:18.406424 14033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.407578 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71ebe1a487514f6b9aec441f39d8ee46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c81e145eb714b6daa5a01c25a02b6c0"
I20260929 00:57:18.407797 13907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:18.411221 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ebe1a487514f6b9aec441f39d8ee46 in term 1.
I20260929 00:57:18.411733 13809 leader_election.cc:302] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c81e145eb714b6daa5a01c25a02b6c0, 71ebe1a487514f6b9aec441f39d8ee46; no voters:
I20260929 00:57:18.413444 14033 leader_election.cc:288] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [CANDIDATE]: Term 1 election: Requested vote from peers 4c81e145eb714b6daa5a01c25a02b6c0 (127.0.97.61:37315), 0c38b6684f0e43afa3611e78b368d8be (127.0.97.60:33377)
I20260929 00:57:18.413944 14033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:18.415601 14033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [term 1 LEADER]: Becoming Leader. State: Replica: 71ebe1a487514f6b9aec441f39d8ee46, State: Running, Role: LEADER
I20260929 00:57:18.416411 14033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } }
I20260929 00:57:18.428224 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71ebe1a487514f6b9aec441f39d8ee46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c38b6684f0e43afa3611e78b368d8be"
I20260929 00:57:18.428519 13981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:18.432133 13981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ebe1a487514f6b9aec441f39d8ee46 in term 1.
I20260929 00:57:18.435357 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ebe1a487514f6b9aec441f39d8ee46. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.435981 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:18.449229 14062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:18.451964 14062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:18.469895 13981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [term 1 FOLLOWER]: Refusing update from remote peer 71ebe1a487514f6b9aec441f39d8ee46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:18.471107 13907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [term 1 FOLLOWER]: Refusing update from remote peer 71ebe1a487514f6b9aec441f39d8ee46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:18.472232 14060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:18.473570 14060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:18.510820 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ebe1a487514f6b9aec441f39d8ee46. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.511134 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.513909 14063 mvcc.cc:205] Tried to move back new op lower bound from 7334475523957235712 to 7334475523765112832. Current Snapshot: MvccSnapshot[applied={T|T < 7334475523957235712}]
I20260929 00:57:18.524734 14064 mvcc.cc:205] Tried to move back new op lower bound from 7334475523957235712 to 7334475523765112832. Current Snapshot: MvccSnapshot[applied={T|T < 7334475523957235712}]
I20260929 00:57:18.526806 14033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.527122 14033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:18.528983 14062 catalog_manager.cc:1347] Generated new cluster ID: de834e02ba6149c0963aabcd80c3fa6a
I20260929 00:57:18.529094 14062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:18.530584 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ebe1a487514f6b9aec441f39d8ee46. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.530911 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.531180 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.531422 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.533854 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.535866 14065 mvcc.cc:205] Tried to move back new op lower bound from 7334475523957235712 to 7334475523765112832. Current Snapshot: MvccSnapshot[applied={T|T < 7334475523957235712}]
I20260929 00:57:18.536053 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:18.549777 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ebe1a487514f6b9aec441f39d8ee46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ebe1a487514f6b9aec441f39d8ee46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43803 } } peers { permanent_uuid: "4c81e145eb714b6daa5a01c25a02b6c0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37315 } } peers { permanent_uuid: "0c38b6684f0e43afa3611e78b368d8be" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33377 } } }
I20260929 00:57:18.550022 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:57:18.609663 14062 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:18.611476 14062 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:18.641816 14062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ebe1a487514f6b9aec441f39d8ee46: Generated new TSK 0
I20260929 00:57:18.642915 14062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:19.184649 14030 catalog_manager.cc:1259] Loaded cluster ID: de834e02ba6149c0963aabcd80c3fa6a
I20260929 00:57:19.184751 14030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: loading cluster ID for follower catalog manager: success
I20260929 00:57:19.187165 14030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: acquiring CA information for follower catalog manager: success
I20260929 00:57:19.189360 14030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c38b6684f0e43afa3611e78b368d8be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:57:19.372021 14058 catalog_manager.cc:1259] Loaded cluster ID: de834e02ba6149c0963aabcd80c3fa6a
I20260929 00:57:19.372128 14058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: loading cluster ID for follower catalog manager: success
I20260929 00:57:19.374807 14058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: acquiring CA information for follower catalog manager: success
I20260929 00:57:19.376672 14058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c81e145eb714b6daa5a01c25a02b6c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:57:19.751207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13791
I20260929 00:57:19.752357 13853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:19.872390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13791
I20260929 00:57:19.902863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13865
I20260929 00:57:19.903955 13927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:20.009068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13865
I20260929 00:57:20.028764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 13939
I20260929 00:57:20.029908 14001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:20.139658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 13939
2026-09-29T00:57:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4801 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:20 dist-test-slave-pzsx krb5kdc[14079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:20 dist-test-slave-pzsx krb5kdc[14079](info): set up 2 sockets
Sep 29 00:57:20 dist-test-slave-pzsx krb5kdc[14079](info): commencing operation
krb5kdc: starting...
W20260929 00:57:21.245157 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:57:21 dist-test-slave-pzsx krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:57:21.479794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:21.810782 14092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:21.811067 14092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:21.811257 14092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:21.818482 14092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:21.818599 14092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:21.818703 14092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:21.818809 14092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:21.818874 14092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:21.828735 14092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:21.830111 14092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:21.831480 14092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:21.844085 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:21.844060 14098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:21.844547 14100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:21.844949 14092 server_base.cc:1034] running on GCE node
I20260929 00:57:21.845785 14092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:21.847262 14092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:21.848546 14092 hybrid_clock.cc:648] HybridClock initialized: now 1790643441848466 us; error 47 us; skew 500 ppm
I20260929 00:57:21.848914 14092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:21 dist-test-slave-pzsx krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:21.851995 14092 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:21.853511 14092 webserver.cc:460] Webserver started at http://127.0.97.62:36499/ using document root <none> and password file <none>
I20260929 00:57:21.854221 14092 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:21.854375 14092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:21.854699 14092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:21.857798 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "78f87c8e6834448c99ceddbae59581d8"
format_stamp: "Formatted at 2026-09-29 00:57:21 on dist-test-slave-pzsx"
server_key: "80710edafec8d6b86b60802963235ede"
server_key_iv: "b0c125a89a4b9cba47386618b460705d"
server_key_version: "encryptionkey@0"
I20260929 00:57:21.858629 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "78f87c8e6834448c99ceddbae59581d8"
format_stamp: "Formatted at 2026-09-29 00:57:21 on dist-test-slave-pzsx"
server_key: "80710edafec8d6b86b60802963235ede"
server_key_iv: "b0c125a89a4b9cba47386618b460705d"
server_key_version: "encryptionkey@0"
I20260929 00:57:21.863952 14092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:57:21.867470 14107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:21.868836 14092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:57:21.869022 14092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "78f87c8e6834448c99ceddbae59581d8"
format_stamp: "Formatted at 2026-09-29 00:57:21 on dist-test-slave-pzsx"
server_key: "80710edafec8d6b86b60802963235ede"
server_key_iv: "b0c125a89a4b9cba47386618b460705d"
server_key_version: "encryptionkey@0"
I20260929 00:57:21.869236 14092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:21.883853 14092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:21.887811 14092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:21.888110 14092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:21.906580 14092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34593
I20260929 00:57:21.906605 14158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34593 every 8 connection(s)
I20260929 00:57:21.908026 14092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:21.911758 14159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:21.912739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14092
I20260929 00:57:21.912957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:57:21.913605 388 external_mini_cluster.cc:1428] Setting key aa5b24f0d4e2fc92414aaa03490974f4
I20260929 00:57:21.926321 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8: Bootstrap starting.
Sep 29 00:57:21 dist-test-slave-pzsx krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:21.929491 14159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:21.930884 14159 log.cc:824] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:21.934530 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8: No bootstrap required, opened a new log
I20260929 00:57:21.938300 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:21.915474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.0.97.62:34593)
0929 00:57:21.916340 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:21.916356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:21.916413 (+ 57us) server_negotiation.cc:407] Connection header received
0929 00:57:21.918435 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:21.918463 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:21.919024 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:21.919497 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:21.920948 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:21.923026 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:21.924407 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:21.925504 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:21.925679 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:21.929861 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:21.929877 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:21.929900 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:21.929976 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:21.934040 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:21.934824 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:21.934839 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:21.934845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:21.935033 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:21.935446 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:21.935451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:21.935456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:21.935962 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:21.936331 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:21.936807 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 00:57:21.937260 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":334,"threads_started":1}
I20260929 00:57:21.942217 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:57:21.942941 14159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } }
I20260929 00:57:21.943310 14159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:21.943413 14159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f87c8e6834448c99ceddbae59581d8, State: Initialized, Role: FOLLOWER
I20260929 00:57:21.943954 14159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } }
I20260929 00:57:21.944201 14159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:21.944337 14159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:21.944557 14159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:21.947032 14159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } }
I20260929 00:57:21.947531 14159 leader_election.cc:302] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f87c8e6834448c99ceddbae59581d8; no voters:
I20260929 00:57:21.948091 14159 leader_election.cc:288] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:21.948227 14164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:21.948956 14164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [term 1 LEADER]: Becoming Leader. State: Replica: 78f87c8e6834448c99ceddbae59581d8, State: Running, Role: LEADER
I20260929 00:57:21.949482 14164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } }
I20260929 00:57:21.950183 14159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:21.953256 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f87c8e6834448c99ceddbae59581d8. Latest consensus state: current_term: 1 leader_uuid: "78f87c8e6834448c99ceddbae59581d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } } }
I20260929 00:57:21.953459 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f87c8e6834448c99ceddbae59581d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f87c8e6834448c99ceddbae59581d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34593 } } }
I20260929 00:57:21.953806 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:21.954007 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:21.961789 14173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:21.967407 14173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:21.976274 14173 catalog_manager.cc:1347] Generated new cluster ID: 88ad5eb9df3d40bca51634c1bf1a1726
I20260929 00:57:21.976430 14173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:21.991420 14173 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:21.992832 14173 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:22.015688 14173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78f87c8e6834448c99ceddbae59581d8: Generated new TSK 0
I20260929 00:57:22.016651 14173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:57:22.054370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14092
I20260929 00:57:22.055558 14154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:22.160979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14092
2026-09-29T00:57:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2024 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:22 dist-test-slave-pzsx krb5kdc[14187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:22 dist-test-slave-pzsx krb5kdc[14187](info): set up 2 sockets
Sep 29 00:57:22 dist-test-slave-pzsx krb5kdc[14187](info): commencing operation
krb5kdc: starting...
W20260929 00:57:23.261053 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:57:23 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:57:23.486166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:23.812861 14200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:23.813162 14200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:23.813359 14200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:23.820865 14200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:23.820979 14200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:23.821091 14200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:23.821234 14200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:23.821328 14200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:23.831265 14200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:23.832667 14200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:23.833992 14200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:23.845573 14205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:23.845633 14208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:23.845638 14206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:23.846715 14200 server_base.cc:1034] running on GCE node
I20260929 00:57:23.847471 14200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:23.848892 14200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:23.850133 14200 hybrid_clock.cc:648] HybridClock initialized: now 1790643443850084 us; error 56 us; skew 500 ppm
I20260929 00:57:23.850498 14200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:23 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:23.858361 14200 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:23.859929 14200 webserver.cc:460] Webserver started at http://127.0.97.62:43049/ using document root <none> and password file <none>
I20260929 00:57:23.860728 14200 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:23.860874 14200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:23.861186 14200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:23.864094 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "0afee6ff1cab4094b1836372bf23c955"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-pzsx"
server_key: "8f557a409ee898c63461400e98f5782c"
server_key_iv: "eb6b0d3f8b09d6dd3a17e58793e1d578"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.864916 14200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "0afee6ff1cab4094b1836372bf23c955"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-pzsx"
server_key: "8f557a409ee898c63461400e98f5782c"
server_key_iv: "eb6b0d3f8b09d6dd3a17e58793e1d578"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.869850 14200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 00:57:23.873132 14215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:23.874408 14200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:23.874589 14200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0afee6ff1cab4094b1836372bf23c955"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-pzsx"
server_key: "8f557a409ee898c63461400e98f5782c"
server_key_iv: "eb6b0d3f8b09d6dd3a17e58793e1d578"
server_key_version: "encryptionkey@0"
I20260929 00:57:23.874778 14200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:23.887979 14200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:23.892372 14200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:23.892688 14200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:23.910813 14200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41677
I20260929 00:57:23.910838 14266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41677 every 8 connection(s)
I20260929 00:57:23.912184 14200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:23.915738 14267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:23.918387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14200
I20260929 00:57:23.918596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:57:23.919421 388 external_mini_cluster.cc:1428] Setting key a57f506ab4c2b2ec1e4b6a24b2df5206
I20260929 00:57:23.928225 14267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap starting.
I20260929 00:57:23.931902 14267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:23.933247 14267 log.cc:824] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:57:23 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:23.936286 14267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: No bootstrap required, opened a new log
I20260929 00:57:23.942691 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:23.921490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.0.97.62:41677)
0929 00:57:23.922298 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:23.922309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:23.922347 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:57:23.923955 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:23.923977 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:23.924361 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:23.924777 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:23.926469 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.927725 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.929584 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:23.930262 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:23.930385 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:23.935466 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:23.935474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:23.935488 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:23.935523 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:23.938661 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.939340 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.939349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.939355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.939536 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:23.939953 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:23.939962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:23.939964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:23.940470 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:23.940844 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:23.941258 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 00:57:23.941694 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":381,"threads_started":1}
I20260929 00:57:23.945013 14267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:23.945401 14267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:23.945523 14267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Initialized, Role: FOLLOWER
I20260929 00:57:23.946148 14267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:23.946346 14267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:23.946463 14267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:23.946586 14267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:23.949000 14267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:23.949447 14267 leader_election.cc:302] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afee6ff1cab4094b1836372bf23c955; no voters:
I20260929 00:57:23.949952 14267 leader_election.cc:288] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:23.950109 14272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:23.951342 14272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 LEADER]: Becoming Leader. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Running, Role: LEADER
I20260929 00:57:23.951830 14272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:23.952090 14267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:23.954690 14274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afee6ff1cab4094b1836372bf23c955. Latest consensus state: current_term: 1 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:23.955065 14274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:23.954787 14273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:23.955245 14273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:23.956796 14278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:23.961531 14278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:23.970988 14278 catalog_manager.cc:1347] Generated new cluster ID: 55952f2c51974276b71e6382faa3f698
I20260929 00:57:23.971088 14278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:23.982836 14278 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:23.984015 14278 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:24.000988 14278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Generated new TSK 0
I20260929 00:57:24.002043 14278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:57:24.064174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:24.392005 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.392287 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.392388 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.399646 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.399744 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.399853 14293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:24.409578 14293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:24.410807 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.412150 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.425066 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.425132 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.425096 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.425999 14293 server_base.cc:1034] running on GCE node
I20260929 00:57:24.426549 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.427381 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.428610 14293 hybrid_clock.cc:648] HybridClock initialized: now 1790643444428564 us; error 64 us; skew 500 ppm
I20260929 00:57:24.428928 14293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:24 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.432035 14293 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:24.433449 14293 webserver.cc:460] Webserver started at http://127.0.97.1:36475/ using document root <none> and password file <none>
I20260929 00:57:24.434096 14293 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.434237 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.434533 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:24.437476 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "eb8e38544712b2c7c88ae023b0d7d83a"
server_key_iv: "5b3b79249fbab79e78484b665b7e396b"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.438232 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "eb8e38544712b2c7c88ae023b0d7d83a"
server_key_iv: "5b3b79249fbab79e78484b665b7e396b"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.443224 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:57:24.446543 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.447778 14293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:24.447947 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "eb8e38544712b2c7c88ae023b0d7d83a"
server_key_iv: "5b3b79249fbab79e78484b665b7e396b"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.448146 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.478576 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.482924 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.483218 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.484125 14293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:24.485649 14293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:24.485751 14293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.485895 14293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:24.485973 14293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.518610 14293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44557
I20260929 00:57:24.518667 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44557 every 8 connection(s)
I20260929 00:57:24.519932 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:24.527427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14293
I20260929 00:57:24.527678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:24.528542 388 external_mini_cluster.cc:1428] Setting key c1a4127e6d3898ede2a0ca099afdf210
Sep 29 00:57:24 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:24.540350 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.522606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47615 (local address 127.0.97.62:41677)
0929 00:57:24.523044 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:24.523049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:24.524040 (+ 991us) server_negotiation.cc:407] Connection header received
0929 00:57:24.525932 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:24.525937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:24.526006 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:24.526105 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:24.527554 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.528794 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.530355 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.530752 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.530904 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.534664 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:24.534669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:24.534673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:24.534702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:24.537328 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.538048 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.538055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.538058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.538180 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:24.538616 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:24.538623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:24.538626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:24.538898 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:24.539050 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:24.539754 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 00:57:24.539986 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260929 00:57:24.541393 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:24.523186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:47615)
0929 00:57:24.523876 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:24.523915 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:24.525649 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:24.526310 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:24.526318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:24.526822 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:24.527377 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.527394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.528950 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.528954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.530193 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:24.530204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:24.530991 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:24.531001 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:24.531250 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:24.531969 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:24.531990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:24.534445 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:24.537474 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.537484 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.537502 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.537929 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.538293 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:24.538308 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:24.538312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:24.538507 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:24.539024 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:24.539032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:24.539398 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 00:57:24.539665 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 00:57:24.540213 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":280,"threads_started":1}
I20260929 00:57:24.543028 14421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:24.543381 14421 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:24.544246 14421 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:24.546805 14232 ts_manager.cc:194] Registered new tserver with Master: 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557)
I20260929 00:57:24.549067 14232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47615
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:57:24.581164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:24.900019 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:24.900309 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:24.900403 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:24.907685 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:24.907789 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:24.907899 14427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:24.917634 14427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:24.918852 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:24.920125 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:24.933379 14433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.933570 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:24.933858 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:24.934136 14427 server_base.cc:1034] running on GCE node
I20260929 00:57:24.934702 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:24.935613 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:24.936834 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790643444936794 us; error 53 us; skew 500 ppm
I20260929 00:57:24.937144 14427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:24 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:24.940080 14427 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:24.941470 14427 webserver.cc:460] Webserver started at http://127.0.97.2:40341/ using document root <none> and password file <none>
I20260929 00:57:24.942134 14427 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:24.942271 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:24.942560 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:24.945400 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "28cf27af00c24e138c7f94a452eaf5ba"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "1d7d21cf0b2d8570ace07c7dfc52c856"
server_key_iv: "dbd57e41b20c1d772a2ce7a8339d5227"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.946163 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "28cf27af00c24e138c7f94a452eaf5ba"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "1d7d21cf0b2d8570ace07c7dfc52c856"
server_key_iv: "dbd57e41b20c1d772a2ce7a8339d5227"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.951100 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:57:24.954770 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.955950 14427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:57:24.956128 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "28cf27af00c24e138c7f94a452eaf5ba"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "1d7d21cf0b2d8570ace07c7dfc52c856"
server_key_iv: "dbd57e41b20c1d772a2ce7a8339d5227"
server_key_version: "encryptionkey@0"
I20260929 00:57:24.956317 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:24.975216 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:24.979193 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:24.979483 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:24.980429 14427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:24.981755 14427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:24.981860 14427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:24.982016 14427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:24.982089 14427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.014349 14427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44299
I20260929 00:57:25.014438 14554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44299 every 8 connection(s)
I20260929 00:57:25.015630 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:25.023491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14427
I20260929 00:57:25.023670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:25.024226 388 external_mini_cluster.cc:1428] Setting key 37570be52107af5a86ca5657d678e27c
Sep 29 00:57:25 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:25.036803 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.018291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41657 (local address 127.0.97.62:41677)
0929 00:57:25.018516 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.018523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.019769 (+ 1246us) server_negotiation.cc:407] Connection header received
0929 00:57:25.021584 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.021590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.021686 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.021773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.023227 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.024319 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.026024 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.026559 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.026675 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.030120 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.030130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.030138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.030185 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.033534 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.034502 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.034505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.034508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.034624 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.035094 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.035097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.035099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.035351 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.035460 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.036345 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.036572 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 00:57:25.038360 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.018888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:41657)
0929 00:57:25.019594 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.019632 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.021291 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.021984 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.021992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.022477 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.023038 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.023077 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.024457 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.024460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.025866 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.025902 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.026770 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.026775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.026959 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.027644 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.027671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.029892 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:25.033696 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:25.033708 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:25.033728 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:25.034380 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:25.034729 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:25.034736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:25.034743 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:25.034979 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:25.035466 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:25.035476 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:25.035949 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.036271 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.036873 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":271,"threads_started":1}
I20260929 00:57:25.039829 14555 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:25.040153 14555 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:25.040946 14555 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:25.042498 14232 ts_manager.cc:194] Registered new tserver with Master: 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2:44299)
I20260929 00:57:25.043817 14232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41657
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:57:25.077991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:25.405359 14560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:25.405644 14560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:25.405804 14560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:25.413275 14560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:25.413386 14560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:25.413606 14560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:25.423393 14560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:25.424796 14560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:25.426153 14560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:25.440057 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:25.440006 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:25.440162 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:25.440935 14560 server_base.cc:1034] running on GCE node
I20260929 00:57:25.441596 14560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:25.442587 14560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:25.443816 14560 hybrid_clock.cc:648] HybridClock initialized: now 1790643445443776 us; error 53 us; skew 500 ppm
I20260929 00:57:25.444142 14560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:25 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:25.447477 14560 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:25.448987 14560 webserver.cc:460] Webserver started at http://127.0.97.3:36877/ using document root <none> and password file <none>
I20260929 00:57:25.449656 14560 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:25.449755 14560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:25.449999 14560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:25.453264 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
format_stamp: "Formatted at 2026-09-29 00:57:25 on dist-test-slave-pzsx"
server_key: "2790a64e2988c839bb08ccda7d6dfb39"
server_key_iv: "14ac70ff30100fd94ebbaf937bd9c230"
server_key_version: "encryptionkey@0"
I20260929 00:57:25.454156 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
format_stamp: "Formatted at 2026-09-29 00:57:25 on dist-test-slave-pzsx"
server_key: "2790a64e2988c839bb08ccda7d6dfb39"
server_key_iv: "14ac70ff30100fd94ebbaf937bd9c230"
server_key_version: "encryptionkey@0"
I20260929 00:57:25.459271 14560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:57:25.462805 14575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.464236 14560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:57:25.464363 14560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
format_stamp: "Formatted at 2026-09-29 00:57:25 on dist-test-slave-pzsx"
server_key: "2790a64e2988c839bb08ccda7d6dfb39"
server_key_iv: "14ac70ff30100fd94ebbaf937bd9c230"
server_key_version: "encryptionkey@0"
I20260929 00:57:25.464596 14560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:25.477425 14560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:25.481686 14560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:25.481921 14560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:25.482785 14560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:25.484033 14560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:25.484103 14560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.484185 14560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:25.484232 14560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:25.517366 14560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35353
I20260929 00:57:25.517463 14687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35353 every 8 connection(s)
I20260929 00:57:25.518993 14560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:25.521382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14560
I20260929 00:57:25.521579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:25.522128 388 external_mini_cluster.cc:1428] Setting key 0dba8c6403a2e2139122e6f05747d113
Sep 29 00:57:25 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:25.539983 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.522008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46907 (local address 127.0.97.62:41677)
0929 00:57:25.522298 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.522305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.523593 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 00:57:25.525835 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.525840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.525930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.526033 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.527460 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.528312 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.530357 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.530723 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.530822 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.534309 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.534313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.534317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.534346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.537076 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.537966 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.537977 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.537979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.538097 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.538478 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.538481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.538482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.538720 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.538823 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.539588 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.539741 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 00:57:25.541033 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.522511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:46907)
0929 00:57:25.523427 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.523493 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.525586 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.526263 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.526272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.526765 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:25.527302 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.527318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.528482 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.528489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.530217 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.530230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.530905 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.530910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.531134 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.531852 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:25.531874 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:25.534105 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:25.537206 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:25.537212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:25.537225 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:25.537832 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:25.538184 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:25.538187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:25.538190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:25.538384 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:25.538840 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:25.538847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:25.539265 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.539485 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.539865 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":375,"threads_started":1}
I20260929 00:57:25.542325 14688 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:25.542642 14688 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:25.543349 14688 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:25.544826 14232 ts_manager.cc:194] Registered new tserver with Master: b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353)
I20260929 00:57:25.545627 14232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46907
I20260929 00:57:25.549995 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:25.553049 14421 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:25.566887 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.554671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.97.62:41677)
0929 00:57:25.554850 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.554862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.555286 (+ 424us) server_negotiation.cc:407] Connection header received
0929 00:57:25.555420 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.555423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.555479 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.555552 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.556914 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.557625 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.558883 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.559286 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.559377 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.561911 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.561916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.561919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.561949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.564753 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.565352 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.565356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.565358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.565464 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.565763 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.565766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.565768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.566068 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.566243 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.566492 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.566659 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:25.570983 14232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:25.578191 14232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:25.603458 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.589662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44557 (local address 127.0.0.1:46438)
0929 00:57:25.590799 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.590850 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.591083 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.592462 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.592471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.592526 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:25.593000 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.593031 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.595872 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.595880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.599177 (+ 3297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.599202 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.601950 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.601960 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.602354 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.602450 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.602597 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.602888 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":2069,"threads_started":1}
I20260929 00:57:25.605772 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.589993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.0.97.1:44557)
0929 00:57:25.591787 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.591797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.591829 (+ 32us) server_negotiation.cc:407] Connection header received
0929 00:57:25.591932 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.591941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.592223 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.593266 (+ 1043us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:25.593557 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.595669 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.599468 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.601652 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.604490 (+ 2838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.604893 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.605026 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.605280 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1614,"thread_start_us":1445,"threads_started":1}
I20260929 00:57:25.611114 14356 tablet_service.cc:1467] Processing CreateTablet for tablet 8c553fc3e05042f69b50945140d7b372 (DEFAULT_TABLE table=test-table [id=6e8ced3148034361826702470d325c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:25.612180 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.590399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.0.97.3:35353)
0929 00:57:25.603713 (+ 13314us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.603726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.603772 (+ 46us) server_negotiation.cc:407] Connection header received
0929 00:57:25.603901 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.603914 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.604292 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.605678 (+ 1386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:25.606040 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.607528 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.610089 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.610987 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.611141 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.611378 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.611676 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.611830 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13059,"thread_start_us":14120,"threads_started":1}
I20260929 00:57:25.612563 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.589883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35353 (local address 127.0.0.1:59870)
0929 00:57:25.592002 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.592020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.592124 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.604849 (+ 12725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.604857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.604898 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:25.605403 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.605417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.607738 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.607746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.609879 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.609893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.611214 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.611221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.611484 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.611511 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.611623 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.611866 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"mutex_wait_us":54,"thread_start_us":800,"threads_started":1}
I20260929 00:57:25.612725 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c553fc3e05042f69b50945140d7b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:25.615056 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.592220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44299 (local address 127.0.0.1:41456)
0929 00:57:25.604106 (+ 11886us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.604134 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.604275 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.607828 (+ 3553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.607833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.607865 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:25.608102 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.608115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.610070 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.610075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.611928 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.611942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.614242 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.614248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.614436 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.614474 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.614583 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.614745 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11713,"mutex_wait_us":50,"thread_start_us":14932,"threads_started":1}
I20260929 00:57:25.615115 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.592608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.97.2:44299)
0929 00:57:25.607162 (+ 14554us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.607171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.607201 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:57:25.607307 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.607316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.607589 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.607826 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:25.608272 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.609879 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.612675 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.614068 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.614202 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.614389 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.614637 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.614814 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14409,"thread_start_us":14276,"threads_started":1}
I20260929 00:57:25.616463 14623 tablet_service.cc:1467] Processing CreateTablet for tablet 8c553fc3e05042f69b50945140d7b372 (DEFAULT_TABLE table=test-table [id=6e8ced3148034361826702470d325c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:25.617733 14623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c553fc3e05042f69b50945140d7b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:25.618969 14490 tablet_service.cc:1467] Processing CreateTablet for tablet 8c553fc3e05042f69b50945140d7b372 (DEFAULT_TABLE table=test-table [id=6e8ced3148034361826702470d325c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:25.620275 14490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c553fc3e05042f69b50945140d7b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:25.627360 14706 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Bootstrap starting.
I20260929 00:57:25.630167 14707 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Bootstrap starting.
I20260929 00:57:25.631934 14706 tablet_bootstrap.cc:654] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:25.633100 14706 log.cc:824] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:25.633622 14707 tablet_bootstrap.cc:654] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:25.635033 14707 log.cc:824] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:25.635584 14709 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Bootstrap starting.
I20260929 00:57:25.638541 14707 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: No bootstrap required, opened a new log
I20260929 00:57:25.638849 14707 ts_tablet_manager.cc:1397] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 00:57:25.639940 14709 tablet_bootstrap.cc:654] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:25.641108 14709 log.cc:824] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:25.641563 14706 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: No bootstrap required, opened a new log
I20260929 00:57:25.641839 14706 ts_tablet_manager.cc:1397] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260929 00:57:25.643666 14709 tablet_bootstrap.cc:492] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: No bootstrap required, opened a new log
I20260929 00:57:25.643929 14709 ts_tablet_manager.cc:1397] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 00:57:25.650185 14707 raft_consensus.cc:348] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.650689 14707 raft_consensus.cc:374] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:25.650785 14707 raft_consensus.cc:729] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f9f0fac2ad4e1980d3886f73be00fd, State: Initialized, Role: FOLLOWER
I20260929 00:57:25.651157 14709 raft_consensus.cc:348] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.651549 14709 raft_consensus.cc:374] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:25.651670 14709 raft_consensus.cc:729] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cf27af00c24e138c7f94a452eaf5ba, State: Initialized, Role: FOLLOWER
I20260929 00:57:25.651517 14707 consensus_queue.cc:260] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.652402 14709 consensus_queue.cc:260] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.652516 14706 raft_consensus.cc:348] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.653018 14706 raft_consensus.cc:374] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:25.653121 14706 raft_consensus.cc:729] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498af2fd881a4bb88b1a63dfc6d2f2c2, State: Initialized, Role: FOLLOWER
I20260929 00:57:25.653793 14706 consensus_queue.cc:260] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.654783 14555 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:25.655517 14709 ts_tablet_manager.cc:1428] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 00:57:25.656400 14688 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:25.657270 14707 ts_tablet_manager.cc:1428] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260929 00:57:25.658988 14706 ts_tablet_manager.cc:1428] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20260929 00:57:25.702147 14713 raft_consensus.cc:482] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:25.702478 14713 raft_consensus.cc:504] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.704203 14713 leader_election.cc:288] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557), b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353)
I20260929 00:57:25.710880 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.704773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34265 (local address 127.0.97.3:35353)
0929 00:57:25.705026 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.705035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.705715 (+ 680us) server_negotiation.cc:407] Connection header received
0929 00:57:25.705910 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.705914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.705969 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.706061 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:25.706660 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.707366 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.709164 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.709916 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.710069 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.710182 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.710494 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.710617 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:57:25.710929 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.704654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34519 (local address 127.0.97.1:44557)
0929 00:57:25.704926 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.704935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.705626 (+ 691us) server_negotiation.cc:407] Connection header received
0929 00:57:25.705846 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.705849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.705907 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.706007 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:25.706607 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.707295 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.709171 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.709935 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.710098 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.710206 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.710551 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.710676 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:57:25.711094 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.704443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44557 (local address 127.0.97.2:34519)
0929 00:57:25.705507 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.705524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.705699 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.706028 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.706032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.706055 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:25.706454 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.706463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.707427 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.707435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.708993 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.709004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.710185 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.710193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.710386 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.710409 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.710515 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.710639 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":48,"thread_start_us":705,"threads_started":1}
I20260929 00:57:25.711450 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.704699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35353 (local address 127.0.97.2:34265)
0929 00:57:25.705604 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:25.705618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:25.705761 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:25.706071 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:25.706076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:25.706096 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:25.706513 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.706521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.707493 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.707501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.709017 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:25.709029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.710151 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.710158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:25.710322 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.710363 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 00:57:25.710463 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:57:25.710596 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":30,"thread_start_us":529,"threads_started":1}
I20260929 00:57:25.712072 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c553fc3e05042f69b50945140d7b372" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" is_pre_election: true
I20260929 00:57:25.712318 14643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c553fc3e05042f69b50945140d7b372" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" is_pre_election: true
I20260929 00:57:25.712615 14376 raft_consensus.cc:2454] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 0.
I20260929 00:57:25.712884 14643 raft_consensus.cc:2454] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 0.
I20260929 00:57:25.713384 14443 leader_election.cc:302] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cf27af00c24e138c7f94a452eaf5ba, 498af2fd881a4bb88b1a63dfc6d2f2c2; no voters:
I20260929 00:57:25.713768 14713 raft_consensus.cc:2781] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:25.713928 14713 raft_consensus.cc:482] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:25.714046 14713 raft_consensus.cc:3037] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:25.716226 14713 raft_consensus.cc:504] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.716917 14713 leader_election.cc:288] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 election: Requested vote from peers 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557), b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353)
I20260929 00:57:25.717290 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c553fc3e05042f69b50945140d7b372" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
I20260929 00:57:25.717381 14643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c553fc3e05042f69b50945140d7b372" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
I20260929 00:57:25.717494 14376 raft_consensus.cc:3037] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:25.717617 14643 raft_consensus.cc:3037] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:25.719765 14376 raft_consensus.cc:2454] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 1.
I20260929 00:57:25.719795 14643 raft_consensus.cc:2454] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 1.
I20260929 00:57:25.720206 14445 leader_election.cc:302] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cf27af00c24e138c7f94a452eaf5ba, b2f9f0fac2ad4e1980d3886f73be00fd; no voters:
I20260929 00:57:25.720521 14713 raft_consensus.cc:2781] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:25.720966 14713 raft_consensus.cc:686] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 LEADER]: Becoming Leader. State: Replica: 28cf27af00c24e138c7f94a452eaf5ba, State: Running, Role: LEADER
I20260929 00:57:25.721444 14713 consensus_queue.cc:237] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } }
I20260929 00:57:25.726084 14231 catalog_manager.cc:5495] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2). New cstate: current_term: 1 leader_uuid: "28cf27af00c24e138c7f94a452eaf5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } health_report { overall_health: HEALTHY } } }
I20260929 00:57:25.755326 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.750203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.97.2:44299)
0929 00:57:25.750421 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.750427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.750447 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:57:25.750551 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.750554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.750624 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.750770 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:25.751120 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.751878 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.753291 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.753723 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.753836 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.754293 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:25.754776 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:25.754931 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.754998 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.755073 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:57:25.763898 14376 raft_consensus.cc:1261] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Refusing update from remote peer 28cf27af00c24e138c7f94a452eaf5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:25.763898 14643 raft_consensus.cc:1261] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Refusing update from remote peer 28cf27af00c24e138c7f94a452eaf5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:25.765038 14717 consensus_queue.cc:1035] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:25.765434 14713 consensus_queue.cc:1035] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 00:57:25.767752 14556 tablet.cc:2369] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:25.770720 14689 tablet.cc:2369] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:25.772472 14422 tablet.cc:2369] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:25.791301 14231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:25.791626 14231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:25.795198 14231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Sending DeleteTablet for 3 replicas of tablet 8c553fc3e05042f69b50945140d7b372
I20260929 00:57:25.796444 14623 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c553fc3e05042f69b50945140d7b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59870
I20260929 00:57:25.796669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14293
I20260929 00:57:25.796775 14490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c553fc3e05042f69b50945140d7b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41456
I20260929 00:57:25.797458 14356 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c553fc3e05042f69b50945140d7b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46438
I20260929 00:57:25.809736 14730 tablet_replica.cc:330] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: stopping tablet replica
I20260929 00:57:25.810189 14730 raft_consensus.cc:2229] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:25.810670 14730 raft_consensus.cc:2258] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:25.813576 14730 ts_tablet_manager.cc:1905] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:25.823158 14730 ts_tablet_manager.cc:1918] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:25.823246 14733 tablet_replica.cc:330] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: stopping tablet replica
I20260929 00:57:25.823354 14730 log.cc:1196] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/8c553fc3e05042f69b50945140d7b372
I20260929 00:57:25.823514 14731 tablet_replica.cc:330] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: stopping tablet replica
I20260929 00:57:25.823783 14733 raft_consensus.cc:2229] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:25.823897 14730 ts_tablet_manager.cc:1939] T 8c553fc3e05042f69b50945140d7b372 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting consensus metadata
I20260929 00:57:25.823937 14731 raft_consensus.cc:2229] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:25.824306 14733 raft_consensus.cc:2258] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:25.824621 14731 raft_consensus.cc:2258] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:25.826064 14218 catalog_manager.cc:4841] TS b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353): tablet 8c553fc3e05042f69b50945140d7b372 (table test-table [id=6e8ced3148034361826702470d325c9d]) successfully deleted
I20260929 00:57:25.826682 14731 ts_tablet_manager.cc:1905] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:25.829809 14733 ts_tablet_manager.cc:1905] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:57:25 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:57:25.834270 14731 ts_tablet_manager.cc:1918] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:25.834447 14731 log.cc:1196] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/8c553fc3e05042f69b50945140d7b372
I20260929 00:57:25.835035 14731 ts_tablet_manager.cc:1939] T 8c553fc3e05042f69b50945140d7b372 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting consensus metadata
I20260929 00:57:25.836962 14216 catalog_manager.cc:4841] TS 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2:44299): tablet 8c553fc3e05042f69b50945140d7b372 (table test-table [id=6e8ced3148034361826702470d325c9d]) successfully deleted
I20260929 00:57:25.837323 14733 ts_tablet_manager.cc:1918] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:25.837514 14733 log.cc:1196] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/8c553fc3e05042f69b50945140d7b372
I20260929 00:57:25.838092 14733 ts_tablet_manager.cc:1939] T 8c553fc3e05042f69b50945140d7b372 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting consensus metadata
I20260929 00:57:25.839948 14216 catalog_manager.cc:4841] TS 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557): tablet 8c553fc3e05042f69b50945140d7b372 (table test-table [id=6e8ced3148034361826702470d325c9d]) successfully deleted
I20260929 00:57:25.841552 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.798024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.97.1:44557)
0929 00:57:25.798265 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.798275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.812896 (+ 14621us) server_negotiation.cc:407] Connection header received
0929 00:57:25.813532 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.813541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.813657 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.813754 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.824802 (+ 11048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.825936 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.828114 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.828805 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.828944 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.834785 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.834801 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.834823 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.834875 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.838675 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.839394 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.839402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.839404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.839544 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.839923 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.839927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.839930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.840316 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.840661 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.840973 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.841181 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:57:25.842231 14416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:25.962581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14293
I20260929 00:57:25.980566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14427
Sep 29 00:57:25 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:57:25.996866 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:25.981401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.0.97.2:44299)
0929 00:57:25.981582 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:25.981588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:25.981638 (+ 50us) server_negotiation.cc:407] Connection header received
0929 00:57:25.981857 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:25.981862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:25.981936 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:25.982054 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:25.983319 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.984154 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.985752 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:25.986207 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:25.986366 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:25.990836 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:25.990851 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:25.990867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:25.990915 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:25.993946 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.994658 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.994665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.994667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.994803 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:25.995210 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:25.995214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:25.995218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:25.995588 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:25.995917 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:25.996166 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 00:57:25.996378 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:57:25.997516 14550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:26.120075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14427
I20260929 00:57:26.138856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14560
Sep 29 00:57:26 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:57:26.155884 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:26.139562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.97.3:35353)
0929 00:57:26.139730 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:26.139735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:26.139754 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:57:26.139929 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:26.139933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:26.139999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:26.140101 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:26.141451 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.142236 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.143566 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:26.143986 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:26.144101 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:26.149031 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:26.149050 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:26.149067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:26.149111 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:26.151941 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:26.152966 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:26.152975 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:26.152977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:26.153155 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:26.153628 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:26.153632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:26.153635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:26.154084 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:26.154426 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:26.155408 (+ 982us) server_negotiation.cc:299] Negotiation successful
0929 00:57:26.155614 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:57:26.156782 14683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:26.248171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14560
I20260929 00:57:26.265439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14200
I20260929 00:57:26.266661 14262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:26.378065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14200
I20260929 00:57:26.399341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--webserver_interface=127.0.97.62
--webserver_port=43049
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:26.731531 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:26.731812 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:26.732000 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:26.739157 14738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:26.739270 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:26.739351 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:26.739434 14738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:26.739497 14738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:26.749344 14738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:26.750733 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:26.752177 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:26.774735 14744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:26.774721 14745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:26.774771 14747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:26.775671 14738 server_base.cc:1034] running on GCE node
I20260929 00:57:26.776685 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:26.778343 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:26.779579 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790643446779531 us; error 62 us; skew 500 ppm
I20260929 00:57:26.779995 14738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:26 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:26.783600 14738 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:26.785368 14738 webserver.cc:460] Webserver started at http://127.0.97.62:43049/ using document root <none> and password file <none>
I20260929 00:57:26.786139 14738 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:26.786296 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:26.792270 14738 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 00:57:26.795320 14754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:26.796669 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:26.796847 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0afee6ff1cab4094b1836372bf23c955"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-pzsx"
server_key: "8f557a409ee898c63461400e98f5782c"
server_key_iv: "eb6b0d3f8b09d6dd3a17e58793e1d578"
server_key_version: "encryptionkey@0"
I20260929 00:57:26.797897 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:26.817263 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:26.821319 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:26.821612 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:26.840801 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41677
I20260929 00:57:26.840823 14805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41677 every 8 connection(s)
I20260929 00:57:26.842398 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:26.843601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14738
I20260929 00:57:26.844646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44557
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=36475
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:26.852991 14806 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:26.856355 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap starting.
I20260929 00:57:26.876231 14806 log.cc:824] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:26.888012 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:26.888671 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap complete.
I20260929 00:57:26.898167 14806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:26.899035 14806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Initialized, Role: FOLLOWER
I20260929 00:57:26.899626 14806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:26.899870 14806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:26.900002 14806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:26.900173 14806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:57:26.904282 14806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:26.904827 14806 leader_election.cc:302] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afee6ff1cab4094b1836372bf23c955; no voters:
I20260929 00:57:26.905558 14806 leader_election.cc:288] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:57:26.905694 14810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:57:26.908011 14806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:26.908777 14810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 LEADER]: Becoming Leader. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Running, Role: LEADER
I20260929 00:57:26.909396 14810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:26.912199 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afee6ff1cab4094b1836372bf23c955. Latest consensus state: current_term: 2 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:26.912599 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:26.913098 14811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:26.913354 14811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:26.918088 14820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:26.928053 14820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:26.929973 14820 catalog_manager.cc:1259] Loaded cluster ID: 55952f2c51974276b71e6382faa3f698
I20260929 00:57:26.930164 14820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:26.934479 14820 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 00:57:26.959465 14820 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 00:57:26.968820 14820 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:26.970613 14820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Loaded TSK: 0
I20260929 00:57:26.972407 14820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:57:27.195808 14808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:27.196089 14808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:27.196235 14808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:27.203909 14808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:27.204018 14808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:27.204185 14808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:27.214689 14808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36475
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:27.215993 14808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:27.217357 14808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:27.232100 14834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:27.232100 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:27.233152 14836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:27.233215 14808 server_base.cc:1034] running on GCE node
I20260929 00:57:27.233948 14808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:27.234931 14808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:27.236164 14808 hybrid_clock.cc:648] HybridClock initialized: now 1790643447236101 us; error 59 us; skew 500 ppm
I20260929 00:57:27.236531 14808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:27 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:27.239722 14808 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:27.241406 14808 webserver.cc:460] Webserver started at http://127.0.97.1:36475/ using document root <none> and password file <none>
I20260929 00:57:27.242092 14808 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:27.242231 14808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:27.247680 14808 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:27.250527 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.252012 14808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:57:27.252182 14808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "eb8e38544712b2c7c88ae023b0d7d83a"
server_key_iv: "5b3b79249fbab79e78484b665b7e396b"
server_key_version: "encryptionkey@0"
I20260929 00:57:27.253216 14808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:27.278793 14808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:27.283314 14808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:27.283615 14808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:27.284724 14808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:27.286087 14808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:27.286224 14808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.286345 14808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:27.286430 14808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.320113 14808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44557
I20260929 00:57:27.320200 14955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44557 every 8 connection(s)
I20260929 00:57:27.321622 14808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:27.330119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14808
I20260929 00:57:27.330870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44299
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=40341
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:27 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:27.346077 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.324468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52185 (local address 127.0.97.62:41677)
0929 00:57:27.325115 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:27.325127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:27.326021 (+ 894us) server_negotiation.cc:407] Connection header received
0929 00:57:27.328085 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:27.328097 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:27.328459 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:27.328840 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:27.330254 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.332278 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.333835 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.334973 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.335179 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.338740 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:27.338747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:27.338761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:27.338797 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:27.341646 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:27.342388 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:27.342395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:27.342400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:27.342565 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:27.342997 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:27.343007 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:27.343009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:27.343453 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:27.343657 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:27.344426 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 00:57:27.344837 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":268,"threads_started":1}
I20260929 00:57:27.346086 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.324678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:52185)
0929 00:57:27.325832 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:27.325894 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:27.327611 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:27.328974 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:27.328981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:27.329483 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:27.330013 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.330030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.332444 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.332447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.333686 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.333697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.335263 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.335268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.335458 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.336149 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:27.336170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:27.338454 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:27.341789 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:27.341797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:27.341812 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:27.342232 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:27.342684 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:27.342689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:27.342693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:27.342873 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:27.343610 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:27.343618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:27.343942 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 00:57:27.344211 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 00:57:27.344773 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":698,"threads_started":1}
I20260929 00:57:27.349385 14956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:27.349690 14956 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:27.350369 14956 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:27.352851 14771 ts_manager.cc:194] Registered new tserver with Master: 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557)
I20260929 00:57:27.355121 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52185
W20260929 00:57:27.663232 14960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:27.663530 14960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:27.663623 14960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:27.670722 14960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:27.670843 14960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:27.670958 14960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:27.680538 14960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40341
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:27.681789 14960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:27.682996 14960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:27.695757 14968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:27.695783 14965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:27.695845 14966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:27.696559 14960 server_base.cc:1034] running on GCE node
I20260929 00:57:27.697206 14960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:27.698124 14960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:27.699317 14960 hybrid_clock.cc:648] HybridClock initialized: now 1790643447699252 us; error 57 us; skew 500 ppm
I20260929 00:57:27.699626 14960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:27 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:27.702900 14960 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:27.704248 14960 webserver.cc:460] Webserver started at http://127.0.97.2:40341/ using document root <none> and password file <none>
I20260929 00:57:27.704963 14960 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:27.705075 14960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:27.710667 14960 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:57:27.713572 14975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.714928 14960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:27.715108 14960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "28cf27af00c24e138c7f94a452eaf5ba"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "1d7d21cf0b2d8570ace07c7dfc52c856"
server_key_iv: "dbd57e41b20c1d772a2ce7a8339d5227"
server_key_version: "encryptionkey@0"
I20260929 00:57:27.716105 14960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:27.742054 14960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:27.746208 14960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:27.746521 14960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:27.747740 14960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:27.749425 14960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:27.749547 14960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.749670 14960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:27.749768 14960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:27.782773 14960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44299
I20260929 00:57:27.782859 15087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44299 every 8 connection(s)
I20260929 00:57:27.784238 14960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:27.784885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 14960
I20260929 00:57:27.785625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35353
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=36877
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:57:27 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:27.804754 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.787132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37897 (local address 127.0.97.62:41677)
0929 00:57:27.787350 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:27.787356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:27.788725 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 00:57:27.790649 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:27.790657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:27.790756 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:27.790880 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:27.792348 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.793289 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.794833 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.795290 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.795439 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.798884 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:27.798890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:27.798893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:27.798926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:27.801617 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:27.802380 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:27.802386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:27.802390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:27.802527 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:27.802960 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:27.802966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:27.802969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:27.803244 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:27.803414 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:27.804111 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 00:57:27.804354 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 00:57:27.805819 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:27.787677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:37897)
0929 00:57:27.788531 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:27.788587 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:27.790366 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:27.791086 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:27.791095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:27.791589 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:27.792155 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.792172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.793450 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.793455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.794673 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:27.794685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:27.795524 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:27.795531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:27.795763 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:27.796457 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:27.796479 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:27.798657 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:27.801782 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:27.801797 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:27.801810 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:27.802246 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:27.802643 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:27.802647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:27.802651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:27.802845 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:27.803389 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:27.803396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:27.803729 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 00:57:27.804016 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 00:57:27.804582 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":354,"threads_started":1}
I20260929 00:57:27.807555 15088 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:27.807932 15088 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:27.808888 15088 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:27.810518 14771 ts_manager.cc:194] Registered new tserver with Master: 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2:44299)
I20260929 00:57:27.811417 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37897
W20260929 00:57:28.136201 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:28.136598 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:28.136792 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:28.144644 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:28.144783 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:28.144958 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:28.155736 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36877
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:28.157290 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:28.158896 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:28.172942 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:28.172992 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:28.172992 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:28.174190 15091 server_base.cc:1034] running on GCE node
I20260929 00:57:28.174929 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:28.176047 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:28.177295 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790643448177230 us; error 63 us; skew 500 ppm
I20260929 00:57:28.177673 15091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:28 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:28.181051 15091 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:28.182660 15091 webserver.cc:460] Webserver started at http://127.0.97.3:36877/ using document root <none> and password file <none>
I20260929 00:57:28.183418 15091 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:28.183597 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:28.189675 15091 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:57:28.192816 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:28.194438 15091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:57:28.194677 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
format_stamp: "Formatted at 2026-09-29 00:57:25 on dist-test-slave-pzsx"
server_key: "2790a64e2988c839bb08ccda7d6dfb39"
server_key_iv: "14ac70ff30100fd94ebbaf937bd9c230"
server_key_version: "encryptionkey@0"
I20260929 00:57:28.195777 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:28.255472 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:28.260251 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:28.260648 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:28.261719 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:28.263084 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:28.263171 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:28.263262 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:28.263316 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:28.300581 15091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35353
I20260929 00:57:28.300648 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35353 every 8 connection(s)
I20260929 00:57:28.302335 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:28.311899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15091
Sep 29 00:57:28 dist-test-slave-pzsx krb5kdc[14187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:28.329648 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.312948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.97.62:41677)
0929 00:57:28.313534 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.313542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.314065 (+ 523us) server_negotiation.cc:407] Connection header received
0929 00:57:28.314275 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.314285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.314349 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.314462 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.316006 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.316863 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.318402 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.318959 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.319114 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.322170 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:28.322176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:28.322179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:28.322222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:28.326234 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.327195 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.327200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.327202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.327422 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.327883 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.327886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.327888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.328400 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:28.328676 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.329014 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.329203 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260929 00:57:28.330363 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.305043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51781 (local address 127.0.97.62:41677)
0929 00:57:28.305257 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.305263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.306656 (+ 1393us) server_negotiation.cc:407] Connection header received
0929 00:57:28.308887 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.308892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.308992 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.309104 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.310774 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.311612 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.313888 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.314336 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.314488 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.318309 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:28.318315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:28.318318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:28.318355 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:28.326184 (+ 7829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.327316 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.327320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.327323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.327433 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.327988 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.327994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.327997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.328393 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:28.328582 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.329551 (+ 969us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.329776 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":57}
I20260929 00:57:28.331454 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.305669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:51781)
0929 00:57:28.306457 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.306505 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.308425 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.309371 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.309380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.309954 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:28.310584 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.310603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.311798 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.311803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.313682 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.313711 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.314608 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.314616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.314868 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.315608 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:28.315633 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:28.318035 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:28.326430 (+ 8395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:28.326442 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:28.326462 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:28.327179 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:28.327561 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:28.327565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:28.327568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:28.327853 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:28.328568 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:28.328576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:28.329061 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.329417 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.330174 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":309,"threads_started":1}
I20260929 00:57:28.333009 15219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41677
I20260929 00:57:28.333345 15219 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:28.334100 15219 heartbeater.cc:511] Master 127.0.97.62:41677 requested a full tablet report, sending...
I20260929 00:57:28.335637 14771 ts_manager.cc:194] Registered new tserver with Master: b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353)
I20260929 00:57:28.336493 14771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51781
I20260929 00:57:28.345002 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:28.352181 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.345939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.97.62:41677)
0929 00:57:28.346105 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.346109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.346632 (+ 523us) server_negotiation.cc:407] Connection header received
0929 00:57:28.346838 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.346841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.346987 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.347125 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:28.347563 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.348318 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.349869 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.350324 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.350434 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.350969 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:28.351610 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:28.351725 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.351854 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.351931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:57:28.358501 14956 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:28.353264 14771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:28.361470 14771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:28.385779 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.373821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44299 (local address 127.0.0.1:41472)
0929 00:57:28.375868 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.375916 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.376098 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.378436 (+ 2338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.378448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.378490 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:28.379030 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.379044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.380857 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.380862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.383182 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.383195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.384699 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.384706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.384917 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.384974 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.385125 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.385342 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"mutex_wait_us":35,"thread_start_us":1638,"threads_started":1}
I20260929 00:57:28.387007 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.374019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.97.2:44299)
0929 00:57:28.377654 (+ 3635us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.377661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.377716 (+ 55us) server_negotiation.cc:407] Connection header received
0929 00:57:28.377844 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.377853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.378148 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.378937 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:28.379167 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.380726 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.383352 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.384467 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.384633 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.384876 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.385196 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.386658 (+ 1462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3486,"thread_start_us":3394,"threads_started":1}
I20260929 00:57:28.387116 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.373915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35353 (local address 127.0.0.1:59880)
0929 00:57:28.378544 (+ 4629us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.378563 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.378736 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.379378 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.379384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.379409 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:28.379759 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.379770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.381722 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.381727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.384072 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.384087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.386310 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.386317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.386491 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.386524 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.386624 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.386759 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3222,"thread_start_us":3083,"threads_started":1}
I20260929 00:57:28.387620 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.377483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.97.1:44557)
0929 00:57:28.378133 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.378141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.378768 (+ 627us) server_negotiation.cc:407] Connection header received
0929 00:57:28.379087 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.379097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.379331 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.379513 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:28.380386 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.381906 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.384477 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.386061 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.386192 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.386439 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.387020 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.387218 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":426,"threads_started":1}
I20260929 00:57:28.387640 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.377793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44557 (local address 127.0.0.1:46448)
0929 00:57:28.378650 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.378673 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.378870 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.379729 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.379737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.379761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:28.380200 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.380214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.382074 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.382080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.384320 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.384330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.386607 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.386613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.386795 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.386829 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.386932 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.387087 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":1,"thread_start_us":509,"threads_started":1}
I20260929 00:57:28.388046 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.374398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.97.3:35353)
0929 00:57:28.376803 (+ 2405us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.376813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.378682 (+ 1869us) server_negotiation.cc:407] Connection header received
0929 00:57:28.378905 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.378912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.379163 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.379366 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:28.379967 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.381549 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.384253 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.385751 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.385884 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.386138 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.387354 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.387535 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2222,"thread_start_us":2018,"threads_started":1}
I20260929 00:57:28.391383 14891 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 (DEFAULT_TABLE table=test-table [id=897c562a666342edb49ac6c56cfea133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:28.391383 15023 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 (DEFAULT_TABLE table=test-table [id=897c562a666342edb49ac6c56cfea133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:28.391953 15154 tablet_service.cc:1467] Processing CreateTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 (DEFAULT_TABLE table=test-table [id=897c562a666342edb49ac6c56cfea133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:28.392926 15023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30a0d8b622452d8a25aaa27fb61623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:28.392951 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30a0d8b622452d8a25aaa27fb61623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:28.393229 15154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f30a0d8b622452d8a25aaa27fb61623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:28.407320 15235 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Bootstrap starting.
I20260929 00:57:28.409696 15236 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Bootstrap starting.
I20260929 00:57:28.410686 15237 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Bootstrap starting.
I20260929 00:57:28.411473 15235 tablet_bootstrap.cc:654] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:28.412875 15235 log.cc:824] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:28.413750 15236 tablet_bootstrap.cc:654] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:28.413780 15237 tablet_bootstrap.cc:654] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:28.414851 15237 log.cc:824] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:28.415058 15236 log.cc:824] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:28.418181 15237 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: No bootstrap required, opened a new log
I20260929 00:57:28.418406 15236 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: No bootstrap required, opened a new log
I20260929 00:57:28.418546 15237 ts_tablet_manager.cc:1397] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 00:57:28.418623 15235 tablet_bootstrap.cc:492] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: No bootstrap required, opened a new log
I20260929 00:57:28.418696 15236 ts_tablet_manager.cc:1397] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 00:57:28.418982 15235 ts_tablet_manager.cc:1397] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 00:57:28.426753 15237 raft_consensus.cc:348] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.427361 15237 raft_consensus.cc:374] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:28.427495 15237 raft_consensus.cc:729] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498af2fd881a4bb88b1a63dfc6d2f2c2, State: Initialized, Role: FOLLOWER
I20260929 00:57:28.428174 15237 consensus_queue.cc:260] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.430210 15237 ts_tablet_manager.cc:1428] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 00:57:28.430186 15236 raft_consensus.cc:348] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.430184 15235 raft_consensus.cc:348] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.430682 15235 raft_consensus.cc:374] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:28.430685 15236 raft_consensus.cc:374] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:28.430796 15236 raft_consensus.cc:729] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cf27af00c24e138c7f94a452eaf5ba, State: Initialized, Role: FOLLOWER
I20260929 00:57:28.430796 15235 raft_consensus.cc:729] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f9f0fac2ad4e1980d3886f73be00fd, State: Initialized, Role: FOLLOWER
I20260929 00:57:28.431495 15235 consensus_queue.cc:260] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.431491 15236 consensus_queue.cc:260] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.433288 15088 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:28.433560 15235 ts_tablet_manager.cc:1428] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260929 00:57:28.433775 15219 heartbeater.cc:503] Master 127.0.97.62:41677 was elected leader, sending a full tablet report...
I20260929 00:57:28.433843 15236 ts_tablet_manager.cc:1428] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260929 00:57:28.536749 15089 tablet.cc:2369] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:28.554301 15220 tablet.cc:2369] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:57:28.574008 14957 tablet.cc:2369] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:28.608575 15242 raft_consensus.cc:482] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:28.608995 15242 raft_consensus.cc:504] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.610821 15242 leader_election.cc:288] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353), 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557)
I20260929 00:57:28.618063 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.611163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40671 (local address 127.0.97.3:35353)
0929 00:57:28.611400 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.611407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.612379 (+ 972us) server_negotiation.cc:407] Connection header received
0929 00:57:28.612741 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.612746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.612826 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.612911 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:28.613468 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.614313 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.616238 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.617108 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.617205 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.617284 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.617679 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.617805 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 00:57:28.618141 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.611184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35353 (local address 127.0.97.2:40671)
0929 00:57:28.612235 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.612259 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.612556 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.612938 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.612945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.612973 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:28.613311 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.613324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.614451 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.614457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.616068 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.616080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.617342 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.617349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.617526 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.617550 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.617622 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.617757 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":652,"threads_started":1}
I20260929 00:57:28.618104 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.611463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47891 (local address 127.0.97.1:44557)
0929 00:57:28.611660 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.611667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.612747 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 00:57:28.612912 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.612917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.612998 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.613089 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:28.613810 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.614631 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.616361 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.617216 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.617365 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.617462 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.617745 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.617880 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:57:28.618402 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.611212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44557 (local address 127.0.97.2:47891)
0929 00:57:28.612658 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:28.612679 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:28.612783 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:28.613212 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:28.613215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:28.613235 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:28.613630 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.613641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.614773 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.614776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.616184 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:28.616200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.617436 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.617441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:28.617571 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.617592 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:57:28.617703 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:57:28.617825 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":1,"thread_start_us":1100,"threads_started":1}
I20260929 00:57:28.619105 15174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30a0d8b622452d8a25aaa27fb61623" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" is_pre_election: true
I20260929 00:57:28.619451 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30a0d8b622452d8a25aaa27fb61623" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" is_pre_election: true
I20260929 00:57:28.619763 15174 raft_consensus.cc:2454] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 0.
I20260929 00:57:28.619961 14911 raft_consensus.cc:2454] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 0.
I20260929 00:57:28.620518 14978 leader_election.cc:302] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cf27af00c24e138c7f94a452eaf5ba, b2f9f0fac2ad4e1980d3886f73be00fd; no voters:
I20260929 00:57:28.620944 15242 raft_consensus.cc:2781] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:28.621097 15242 raft_consensus.cc:482] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:28.621212 15242 raft_consensus.cc:3037] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:28.623319 15242 raft_consensus.cc:504] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.623905 15242 leader_election.cc:288] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 election: Requested vote from peers b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353), 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557)
I20260929 00:57:28.624389 15174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30a0d8b622452d8a25aaa27fb61623" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
I20260929 00:57:28.624389 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f30a0d8b622452d8a25aaa27fb61623" candidate_uuid: "28cf27af00c24e138c7f94a452eaf5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
I20260929 00:57:28.624615 14911 raft_consensus.cc:3037] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:28.624615 15174 raft_consensus.cc:3037] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:28.627768 14911 raft_consensus.cc:2454] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 1.
I20260929 00:57:28.627768 15174 raft_consensus.cc:2454] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cf27af00c24e138c7f94a452eaf5ba in term 1.
I20260929 00:57:28.628196 14976 leader_election.cc:302] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28cf27af00c24e138c7f94a452eaf5ba, 498af2fd881a4bb88b1a63dfc6d2f2c2, b2f9f0fac2ad4e1980d3886f73be00fd; no voters:
I20260929 00:57:28.628602 15242 raft_consensus.cc:2781] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:28.629053 15242 raft_consensus.cc:686] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 LEADER]: Becoming Leader. State: Replica: 28cf27af00c24e138c7f94a452eaf5ba, State: Running, Role: LEADER
I20260929 00:57:28.629575 15242 consensus_queue.cc:237] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } }
I20260929 00:57:28.633913 14771 catalog_manager.cc:5495] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2). New cstate: current_term: 1 leader_uuid: "28cf27af00c24e138c7f94a452eaf5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cf27af00c24e138c7f94a452eaf5ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 } health_report { overall_health: UNKNOWN } } }
I20260929 00:57:28.681648 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.676379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.0.97.2:44299)
0929 00:57:28.676629 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.676635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.676653 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:57:28.676742 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.676745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.676829 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.676982 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:28.677355 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.678057 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.679468 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.679884 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.680018 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.680551 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:28.681052 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:28.681211 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.681277 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.681355 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:28.689713 15174 raft_consensus.cc:1261] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Refusing update from remote peer 28cf27af00c24e138c7f94a452eaf5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:28.689713 14911 raft_consensus.cc:1261] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Refusing update from remote peer 28cf27af00c24e138c7f94a452eaf5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:28.690784 15246 consensus_queue.cc:1035] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f9f0fac2ad4e1980d3886f73be00fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:28.691156 15242 consensus_queue.cc:1035] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:28.703537 15249 mvcc.cc:205] Tried to move back new op lower bound from 7334475565822709760 to 7334475565590622208. Current Snapshot: MvccSnapshot[applied={T|T < 7334475565822709760}]
I20260929 00:57:28.722107 14771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:28.722399 14771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:28.725759 14771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Sending DeleteTablet for 3 replicas of tablet 0f30a0d8b622452d8a25aaa27fb61623
I20260929 00:57:28.726994 15154 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59880
I20260929 00:57:28.726994 15023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260929 00:57:28.727319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14808
I20260929 00:57:28.727764 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f30a0d8b622452d8a25aaa27fb61623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46448
I20260929 00:57:28.727970 15259 tablet_replica.cc:330] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: stopping tablet replica
I20260929 00:57:28.728449 15259 raft_consensus.cc:2229] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:28.728691 15260 tablet_replica.cc:330] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: stopping tablet replica
I20260929 00:57:28.729146 15260 raft_consensus.cc:2229] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:28.729321 15259 raft_consensus.cc:2258] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:28.729797 15260 raft_consensus.cc:2258] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:28.730104 15258 tablet_replica.cc:330] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: stopping tablet replica
I20260929 00:57:28.730525 15258 raft_consensus.cc:2229] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:28.731045 15258 raft_consensus.cc:2258] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:28.732846 15260 ts_tablet_manager.cc:1905] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:28.733008 15258 ts_tablet_manager.cc:1905] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:28.739879 15258 ts_tablet_manager.cc:1918] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:28.739888 15260 ts_tablet_manager.cc:1918] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:28.740051 15260 log.cc:1196] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/0f30a0d8b622452d8a25aaa27fb61623
I20260929 00:57:28.740195 15258 log.cc:1196] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/0f30a0d8b622452d8a25aaa27fb61623
I20260929 00:57:28.740813 15258 ts_tablet_manager.cc:1939] T 0f30a0d8b622452d8a25aaa27fb61623 P b2f9f0fac2ad4e1980d3886f73be00fd: Deleting consensus metadata
I20260929 00:57:28.741011 15260 ts_tablet_manager.cc:1939] T 0f30a0d8b622452d8a25aaa27fb61623 P 498af2fd881a4bb88b1a63dfc6d2f2c2: Deleting consensus metadata
I20260929 00:57:28.742776 14757 catalog_manager.cc:4841] TS b2f9f0fac2ad4e1980d3886f73be00fd (127.0.97.3:35353): tablet 0f30a0d8b622452d8a25aaa27fb61623 (table test-table [id=897c562a666342edb49ac6c56cfea133]) successfully deleted
I20260929 00:57:28.742753 14755 catalog_manager.cc:4841] TS 498af2fd881a4bb88b1a63dfc6d2f2c2 (127.0.97.1:44557): tablet 0f30a0d8b622452d8a25aaa27fb61623 (table test-table [id=897c562a666342edb49ac6c56cfea133]) successfully deleted
I20260929 00:57:28.745939 15259 ts_tablet_manager.cc:1905] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:28.747124 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.729489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.97.1:44557)
0929 00:57:28.729653 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.729658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.729677 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:57:28.729734 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.729739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.729839 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.729951 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.732053 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.733216 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.735324 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.735978 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.736103 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.739688 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:28.739704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:28.739727 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:28.739784 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:28.744175 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.745005 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.745011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.745014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.745138 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.745563 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.745566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.745568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.745884 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:28.746209 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.746525 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.746745 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:28.747779 14951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:28.754037 15259 ts_tablet_manager.cc:1918] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:28.754217 15259 log.cc:1196] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/0f30a0d8b622452d8a25aaa27fb61623
I20260929 00:57:28.754781 15259 ts_tablet_manager.cc:1939] T 0f30a0d8b622452d8a25aaa27fb61623 P 28cf27af00c24e138c7f94a452eaf5ba: Deleting consensus metadata
I20260929 00:57:28.756659 14755 catalog_manager.cc:4841] TS 28cf27af00c24e138c7f94a452eaf5ba (127.0.97.2:44299): tablet 0f30a0d8b622452d8a25aaa27fb61623 (table test-table [id=897c562a666342edb49ac6c56cfea133]) successfully deleted
I20260929 00:57:28.856271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14808
I20260929 00:57:28.874827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14960
I20260929 00:57:28.895161 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:28.875886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.97.2:44299)
0929 00:57:28.876126 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:28.876133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:28.876181 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:57:28.876261 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:28.876267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:28.876365 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:28.876478 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:28.877774 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.879025 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.880300 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:28.881037 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:28.881170 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:28.887811 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:28.887832 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:28.887854 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:28.887923 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:28.892097 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.892952 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.892962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.892965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.893136 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:28.893482 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:28.893487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:28.893491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:28.893981 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:28.894307 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:28.894480 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 00:57:28.894704 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 00:57:28.896194 15083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:29.010690 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14960
I20260929 00:57:29.029233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15091
I20260929 00:57:29.041893 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:29.029948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.0.97.3:35353)
0929 00:57:29.030139 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:29.030144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:29.030162 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:57:29.030281 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:29.030285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:29.030356 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:29.030441 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:29.031782 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.032489 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.033858 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:29.034238 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:29.034337 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:29.036848 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:29.036867 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:29.036884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:29.036914 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:29.039411 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:29.040031 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:29.040038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:29.040040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:29.040180 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:29.040475 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:29.040478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:29.040480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:29.040920 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:29.041142 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:29.041341 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 00:57:29.041491 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 00:57:29.042559 15214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:29.134272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15091
I20260929 00:57:29.151291 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 14738
I20260929 00:57:29.152563 14801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:29.265885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 14738
I20260929 00:57:29.287791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--webserver_interface=127.0.97.62
--webserver_port=43049
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:29.613567 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:29.613921 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:29.614079 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:29.621243 15265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:29.621387 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:29.621497 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:29.621609 15265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:29.621697 15265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:29.631443 15265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:29.632810 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:29.634088 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:29.645915 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:29.645884 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:29.645987 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:29.650599 15265 server_base.cc:1034] running on GCE node
I20260929 00:57:29.651386 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:29.652895 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:29.654112 15265 hybrid_clock.cc:648] HybridClock initialized: now 1790643449654072 us; error 32 us; skew 500 ppm
I20260929 00:57:29.654471 15265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:29 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:29.657466 15265 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:29.658978 15265 webserver.cc:460] Webserver started at http://127.0.97.62:43049/ using document root <none> and password file <none>
I20260929 00:57:29.659694 15265 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:29.659839 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:29.665179 15265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:57:29.667865 15280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:29.669190 15265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:57:29.669396 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0afee6ff1cab4094b1836372bf23c955"
format_stamp: "Formatted at 2026-09-29 00:57:23 on dist-test-slave-pzsx"
server_key: "8f557a409ee898c63461400e98f5782c"
server_key_iv: "eb6b0d3f8b09d6dd3a17e58793e1d578"
server_key_version: "encryptionkey@0"
I20260929 00:57:29.670356 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:29.700390 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:29.704278 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:29.704588 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:29.722932 15265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41677
I20260929 00:57:29.722962 15331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41677 every 8 connection(s)
I20260929 00:57:29.724408 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:29.730883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15265
I20260929 00:57:29.731011 15332 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:57:29.731679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44557
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=36475
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:57:29.734290 15332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap starting.
I20260929 00:57:29.754522 15332 log.cc:824] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:29.770186 15332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:57:29.770706 15332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955: Bootstrap complete.
I20260929 00:57:29.782212 15332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:29.783344 15332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Initialized, Role: FOLLOWER
I20260929 00:57:29.784030 15332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:29.784286 15332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:29.784417 15332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:29.784641 15332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:57:29.788286 15332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:29.788825 15332 leader_election.cc:302] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afee6ff1cab4094b1836372bf23c955; no voters:
I20260929 00:57:29.789491 15332 leader_election.cc:288] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 00:57:29.789626 15336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:57:29.790894 15336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [term 3 LEADER]: Becoming Leader. State: Replica: 0afee6ff1cab4094b1836372bf23c955, State: Running, Role: LEADER
I20260929 00:57:29.791465 15336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } }
I20260929 00:57:29.791683 15332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:29.793926 15338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afee6ff1cab4094b1836372bf23c955. Latest consensus state: current_term: 3 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:29.794312 15338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:29.794835 15337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0afee6ff1cab4094b1836372bf23c955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afee6ff1cab4094b1836372bf23c955" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41677 } } }
I20260929 00:57:29.795097 15337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afee6ff1cab4094b1836372bf23c955 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:29.798548 15342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:29.807708 15342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:29.809331 15342 catalog_manager.cc:1259] Loaded cluster ID: 55952f2c51974276b71e6382faa3f698
I20260929 00:57:29.809504 15342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 00:57:29.811705 15342 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ce1e1d9ad google::LogMessage::Fail() at ??:0
@ 0x7f6ce1e2088a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ce1e1d4de google::LogMessage::Flush() at ??:0
@ 0x7f6ce1e1ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6cf7850512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6cf784d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6cf78d5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6cf78d5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6ce4a69345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ce4a6c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6ce4a6c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6ce4a47eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ce7d216db start_thread at ??:0
@ 0x7f6ce0cb761f clone at ??:0
W20260929 00:57:30.112815 15334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:30.113121 15334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:30.113284 15334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:30.120491 15334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:30.120697 15334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:30.120867 15334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:30.130412 15334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36475
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:30.131675 15334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:30.133052 15334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:30.146241 15371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:30.146325 15372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:30.146272 15374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:30.147075 15334 server_base.cc:1034] running on GCE node
I20260929 00:57:30.147646 15334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:30.148566 15334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:30.149798 15334 hybrid_clock.cc:648] HybridClock initialized: now 1790643450149750 us; error 42 us; skew 500 ppm
I20260929 00:57:30.150110 15334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:30 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:30.153473 15334 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:30.154968 15334 webserver.cc:460] Webserver started at http://127.0.97.1:36475/ using document root <none> and password file <none>
I20260929 00:57:30.155670 15334 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:30.155813 15334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:30.161058 15334 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:57:30.165932 15381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.167259 15334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:30.167469 15334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "498af2fd881a4bb88b1a63dfc6d2f2c2"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "eb8e38544712b2c7c88ae023b0d7d83a"
server_key_iv: "5b3b79249fbab79e78484b665b7e396b"
server_key_version: "encryptionkey@0"
I20260929 00:57:30.168438 15334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:30.187104 15334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:30.191331 15334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:30.191610 15334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:30.192606 15334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:30.193893 15334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:30.194008 15334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.194133 15334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:30.194223 15334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.227214 15334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44557
I20260929 00:57:30.227264 15493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44557 every 8 connection(s)
I20260929 00:57:30.228678 15334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:30.233203 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.231422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:55729)
0929 00:57:30.232123 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.232311 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":275,"threads_started":1}
W20260929 00:57:30.233817 15494 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41677 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41677: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
I20260929 00:57:30.234544 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.234257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:58797)
0929 00:57:30.234384 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.234447 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 00:57:30.235286 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.235016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:33507)
0929 00:57:30.235125 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.235190 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 00:57:30.235529 15494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:57:30.236357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15334
I20260929 00:57:30.236967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44299
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=40341
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:30.560858 15497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:30.561132 15497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:30.561300 15497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:30.568388 15497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:30.568549 15497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:30.568744 15497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:30.578722 15497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40341
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:30.580008 15497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:30.581357 15497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:30.593973 15502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:30.594026 15503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:30.593930 15505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:30.595116 15497 server_base.cc:1034] running on GCE node
I20260929 00:57:30.595700 15497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:30.596678 15497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:30.597896 15497 hybrid_clock.cc:648] HybridClock initialized: now 1790643450597842 us; error 65 us; skew 500 ppm
I20260929 00:57:30.598202 15497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:30 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:30.601152 15497 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:30.602604 15497 webserver.cc:460] Webserver started at http://127.0.97.2:40341/ using document root <none> and password file <none>
I20260929 00:57:30.603266 15497 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:30.603408 15497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:30.608464 15497 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:57:30.611079 15512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.612342 15497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:30.612622 15497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "28cf27af00c24e138c7f94a452eaf5ba"
format_stamp: "Formatted at 2026-09-29 00:57:24 on dist-test-slave-pzsx"
server_key: "1d7d21cf0b2d8570ace07c7dfc52c856"
server_key_iv: "dbd57e41b20c1d772a2ce7a8339d5227"
server_key_version: "encryptionkey@0"
I20260929 00:57:30.613675 15497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:30.637831 15497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:30.641693 15497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:30.641975 15497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:30.642943 15497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:30.644263 15497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:30.644394 15497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.644547 15497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:30.644637 15497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:30.677455 15497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44299
I20260929 00:57:30.677546 15624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44299 every 8 connection(s)
I20260929 00:57:30.678910 15497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:30.683611 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.681861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:55081)
0929 00:57:30.682562 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.682773 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":280,"threads_started":1}
W20260929 00:57:30.684219 15625 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41677 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41677: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
I20260929 00:57:30.685039 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.684726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:50113)
0929 00:57:30.684871 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.684942 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 00:57:30.685810 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:30.685507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:55717)
0929 00:57:30.685638 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:30.685712 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 00:57:30.686041 15625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:57:30.688767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15497
I20260929 00:57:30.689360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35353
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41677
--webserver_port=36877
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:31.005266 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:31.005568 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:31.005664 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:31.012730 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:31.012836 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:31.012946 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:31.022681 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36877
--tserver_master_addrs=127.0.97.62:41677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:31.023895 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:31.025139 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:31.038158 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:31.038192 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:31.038126 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:31.038897 15628 server_base.cc:1034] running on GCE node
I20260929 00:57:31.039554 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:31.040459 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:31.041712 15628 hybrid_clock.cc:648] HybridClock initialized: now 1790643451041657 us; error 49 us; skew 500 ppm
I20260929 00:57:31.042016 15628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:31 dist-test-slave-pzsx krb5kdc[14187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:31.044916 15628 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:31.046340 15628 webserver.cc:460] Webserver started at http://127.0.97.3:36877/ using document root <none> and password file <none>
I20260929 00:57:31.046993 15628 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:31.047145 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:31.052474 15628 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:57:31.055019 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:31.056375 15628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:57:31.056592 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "b2f9f0fac2ad4e1980d3886f73be00fd"
format_stamp: "Formatted at 2026-09-29 00:57:25 on dist-test-slave-pzsx"
server_key: "2790a64e2988c839bb08ccda7d6dfb39"
server_key_iv: "14ac70ff30100fd94ebbaf937bd9c230"
server_key_version: "encryptionkey@0"
I20260929 00:57:31.057575 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:31.070025 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:31.074373 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:31.074653 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:31.075585 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:31.076967 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:31.077097 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:31.077260 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:31.077382 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:31.111373 15628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35353
I20260929 00:57:31.111472 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35353 every 8 connection(s)
I20260929 00:57:31.112897 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:31.118153 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.116045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:56127)
0929 00:57:31.117048 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.117287 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":451,"threads_started":1}
W20260929 00:57:31.118794 15756 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41677 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41677: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
I20260929 00:57:31.119594 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.119281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:38593)
0929 00:57:31.119420 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.119496 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 00:57:31.120352 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.120048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:47321)
0929 00:57:31.120176 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.120255 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 00:57:31.120626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15628
W20260929 00:57:31.120637 15756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:57:31.236857 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.236041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:59805)
0929 00:57:31.236484 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.236636 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
I20260929 00:57:31.687268 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:31.686554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:36319)
0929 00:57:31.686965 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:31.687044 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":206,"threads_started":1}
I20260929 00:57:32.121855 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:32.121112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:38919)
0929 00:57:32.121554 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:32.121634 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260929 00:57:32.238569 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:32.237726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:38117)
0929 00:57:32.238287 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:32.238407 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":291,"threads_started":1}
I20260929 00:57:32.689004 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:32.688160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:51947)
0929 00:57:32.688746 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:32.688822 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":341,"threads_started":1}
I20260929 00:57:33.123297 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.122634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:41949)
0929 00:57:33.123059 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.123149 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260929 00:57:33.240067 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.239409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:41173)
0929 00:57:33.239814 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.239898 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":217,"threads_started":1}
I20260929 00:57:33.690534 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:33.689836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:54641)
0929 00:57:33.690276 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:33.690363 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260929 00:57:34.124883 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.124154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:38407)
0929 00:57:34.124630 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.124707 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":214,"threads_started":1}
I20260929 00:57:34.241731 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.241003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:54095)
0929 00:57:34.241450 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.241531 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":218,"threads_started":1}
I20260929 00:57:34.692133 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:34.691423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:55525)
0929 00:57:34.691845 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:34.691933 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260929 00:57:35.126314 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.125645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:42203)
0929 00:57:35.126061 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.126132 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260929 00:57:35.243433 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.242671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:42627)
0929 00:57:35.243145 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.243228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260929 00:57:35.693735 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:35.693054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:60781)
0929 00:57:35.693492 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:35.693572 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260929 00:57:36.127849 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.127163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:45541)
0929 00:57:36.127596 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.127672 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260929 00:57:36.244962 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.244252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:50797)
0929 00:57:36.244713 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.244794 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260929 00:57:36.695210 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:36.694522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:54197)
0929 00:57:36.694957 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:36.695061 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260929 00:57:37.129444 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.128749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:55493)
0929 00:57:37.129167 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.129248 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260929 00:57:37.246536 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.245815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:43877)
0929 00:57:37.246255 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.246329 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":201,"threads_started":1}
I20260929 00:57:37.697026 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:37.696261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:60555)
0929 00:57:37.696746 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:37.696821 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260929 00:57:38.131112 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.130365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:51603)
0929 00:57:38.130826 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.130902 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":216,"threads_started":1}
I20260929 00:57:38.248073 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.247415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:43055)
0929 00:57:38.247843 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.247921 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260929 00:57:38.698686 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:38.697989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:47799)
0929 00:57:38.698433 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:38.698526 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
I20260929 00:57:39.132565 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.131855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:33079)
0929 00:57:39.132276 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.132348 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260929 00:57:39.249806 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.249058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:43469)
0929 00:57:39.249546 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.249634 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":237,"threads_started":1}
I20260929 00:57:39.700373 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:39.699602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:60889)
0929 00:57:39.700112 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:39.700192 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260929 00:57:40.134121 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.133393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:60557)
0929 00:57:40.133881 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.133955 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
I20260929 00:57:40.251386 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.250691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:33241)
0929 00:57:40.251151 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.251228 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260929 00:57:40.705204 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:40.701248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:36457)
0929 00:57:40.704961 (+ 3713us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:40.705040 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3617,"thread_start_us":3508,"threads_started":1}
I20260929 00:57:41.135609 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.134977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:43491)
0929 00:57:41.135378 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.135455 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":221,"threads_started":1}
I20260929 00:57:41.253223 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.252376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:35679)
0929 00:57:41.252927 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.253019 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":322,"threads_started":1}
I20260929 00:57:41.706859 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:41.706139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:52789)
0929 00:57:41.706566 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:41.706657 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260929 00:57:42.137312 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:42.136614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:33897)
0929 00:57:42.137062 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:42.137140 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260929 00:57:42.254880 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:42.254175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:52837)
0929 00:57:42.254616 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:42.254701 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260929 00:57:42.708607 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:42.707829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:52603)
0929 00:57:42.708259 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:42.708344 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":235,"threads_started":1}
I20260929 00:57:43.139002 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.138193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:40133)
0929 00:57:43.138765 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:43.138841 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":344,"threads_started":1}
I20260929 00:57:43.256783 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.255829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:46139)
0929 00:57:43.256279 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:43.256372 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":217,"threads_started":1}
I20260929 00:57:43.710234 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:43.709511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:49245)
0929 00:57:43.709962 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:43.710047 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":211,"threads_started":1}
I20260929 00:57:44.140712 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.139867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:49999)
0929 00:57:44.140322 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.140403 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":221,"threads_started":1}
I20260929 00:57:44.258468 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.257724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:42233)
0929 00:57:44.258171 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.258260 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260929 00:57:44.711819 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:44.711125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:34463)
0929 00:57:44.711559 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:44.711639 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":231,"threads_started":1}
I20260929 00:57:45.142293 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:45.141550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:51277)
0929 00:57:45.142029 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:45.142114 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":228,"threads_started":1}
I20260929 00:57:45.260237 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:45.259540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.1:34193)
0929 00:57:45.259948 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:45.260044 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260929 00:57:45.713778 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:45.713050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.2:39449)
0929 00:57:45.713488 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:45.713586 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260929 00:57:46.129297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15334
I20260929 00:57:46.144258 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:46.143451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41677 (local address 127.0.97.3:37963)
0929 00:57:46.143972 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:46.144072 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
I20260929 00:57:46.154023 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:46.130590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.0.97.1:44557)
0929 00:57:46.131698 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:46.131720 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:46.131783 (+ 63us) server_negotiation.cc:407] Connection header received
0929 00:57:46.133955 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:46.133967 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:46.134399 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:46.134691 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:46.135993 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.137279 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.138487 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.139122 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.139249 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:46.148098 (+ 8849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:46.148107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:46.148126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:46.148165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:46.151082 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.151737 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.151743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.151751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.151911 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.152227 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.152230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.152231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.152835 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:46.153044 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:46.153409 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 00:57:46.153748 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":701,"threads_started":1}
I20260929 00:57:46.155416 15489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:46.238511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15334
I20260929 00:57:46.256142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15497
I20260929 00:57:46.273319 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:46.257095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.97.2:44299)
0929 00:57:46.257719 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:46.257733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:46.257772 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:57:46.259324 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:46.259334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:46.259767 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:46.260080 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:46.261499 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.262792 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.264087 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.264806 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.264991 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:46.267648 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:46.267656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:46.267671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:46.267705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:46.270501 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.271113 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.271118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.271125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.271278 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.271584 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.271587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.271589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.272090 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:46.272306 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:46.272701 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 00:57:46.273028 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":337,"threads_started":1}
I20260929 00:57:46.274684 15620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:46.356882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15497
I20260929 00:57:46.373641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15628
I20260929 00:57:46.391417 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:46.374529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.97.3:35353)
0929 00:57:46.375143 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:46.375158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:46.375199 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:57:46.376923 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:46.376932 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:46.377360 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:46.377749 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:46.379210 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.380423 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.381801 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:46.382460 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:46.382672 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:46.385331 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:46.385338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:46.385353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:46.385394 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:46.388133 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.388899 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.388906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.388912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.389074 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:46.389465 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:46.389470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:46.389475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:46.389975 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:46.390279 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:46.390721 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 00:57:46.391116 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":248,"threads_started":1}
I20260929 00:57:46.392817 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:46.476104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15628
2026-09-29T00:57:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24308 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:57:46 dist-test-slave-pzsx krb5kdc[15813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:57:46 dist-test-slave-pzsx krb5kdc[15813](info): set up 2 sockets
Sep 29 00:57:46 dist-test-slave-pzsx krb5kdc[15813](info): commencing operation
krb5kdc: starting...
W20260929 00:57:47.568967 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:57:47 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:57:47.800760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:48.125231 15826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:48.125515 15826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:48.125699 15826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:48.132858 15826 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:57:48.132949 15826 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:57:48.133045 15826 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:57:48.133144 15826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:57:48.133234 15826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:48.133350 15826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:48.133416 15826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:57:48.133512 15826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:57:48.144594 15826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:48.145929 15826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:48.147240 15826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:48.159365 15831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:48.159392 15832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:48.159469 15834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:48.159775 15826 server_base.cc:1034] running on GCE node
I20260929 00:57:48.160832 15826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:48.162261 15826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:48.163481 15826 hybrid_clock.cc:648] HybridClock initialized: now 1790643468163418 us; error 32 us; skew 500 ppm
I20260929 00:57:48.163822 15826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:48 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:48.166986 15826 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:57:48.168383 15826 webserver.cc:460] Webserver started at http://127.0.97.62:33859/ using document root <none> and password file <none>
I20260929 00:57:48.169186 15826 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:48.169325 15826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:48.169646 15826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:48.172703 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "0c90c3c984b1424181f61293beb75439"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "0bc1fcacc668aa2b3ff41ad81a2d0e59"
server_key_iv: "2230176c891faa60b3b77e93ca7b3b0d"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.173493 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c90c3c984b1424181f61293beb75439"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "0bc1fcacc668aa2b3ff41ad81a2d0e59"
server_key_iv: "2230176c891faa60b3b77e93ca7b3b0d"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.178721 15826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:57:48.182248 15841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:48.183527 15826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:57:48.183707 15826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0c90c3c984b1424181f61293beb75439"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "0bc1fcacc668aa2b3ff41ad81a2d0e59"
server_key_iv: "2230176c891faa60b3b77e93ca7b3b0d"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.183919 15826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:48.201828 15826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:48.205763 15826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:48.206046 15826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:48.224562 15826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43469
I20260929 00:57:48.224650 15892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43469 every 8 connection(s)
I20260929 00:57:48.225950 15826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:57:48.229666 15893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:48.233060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15826
I20260929 00:57:48.233294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:57:48.234090 388 external_mini_cluster.cc:1428] Setting key 21ebd686ec42800115de30f230072473
I20260929 00:57:48.242887 15893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap starting.
I20260929 00:57:48.246978 15893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:48.248396 15893 log.cc:824] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:57:48 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:48.251629 15893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: No bootstrap required, opened a new log
I20260929 00:57:48.257941 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:48.236160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.0.97.62:43469)
0929 00:57:48.236977 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:48.236989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:48.237030 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:57:48.238665 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:48.238688 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:48.239094 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:48.239427 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:48.241357 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.242631 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.244570 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.245246 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.245373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:48.250452 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:48.250460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:48.250481 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:48.250521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:48.253716 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:48.254562 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:48.254569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:48.254575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:48.254739 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:48.255186 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:48.255189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:48.255191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:48.255725 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:48.255930 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:48.256413 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 00:57:48.256883 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":423,"threads_started":1}
I20260929 00:57:48.260694 15893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:57:48.261058 15893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:48.261173 15893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Initialized, Role: FOLLOWER
I20260929 00:57:48.261718 15893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:57:48.261950 15893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:57:48.262079 15893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:57:48.262260 15893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:48.264456 15893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:57:48.265034 15893 leader_election.cc:302] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c90c3c984b1424181f61293beb75439; no voters:
I20260929 00:57:48.265697 15893 leader_election.cc:288] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:57:48.265830 15898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:48.266597 15898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 LEADER]: Becoming Leader. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Running, Role: LEADER
I20260929 00:57:48.267153 15898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:57:48.267992 15893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:57:48.270308 15900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c90c3c984b1424181f61293beb75439. Latest consensus state: current_term: 1 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:57:48.270285 15899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:57:48.270712 15900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:48.270743 15899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:57:48.272969 15904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:57:48.278462 15904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:57:48.286962 15904 catalog_manager.cc:1347] Generated new cluster ID: 4aad156ec0b2404ab2fc5bb48a29609a
I20260929 00:57:48.287143 15904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:57:48.300401 15904 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:57:48.302474 15904 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:57:48.319654 15904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Generated new TSK 0
I20260929 00:57:48.320937 15904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:57:48.385219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:48.716264 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:48.716622 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:48.716797 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:48.724130 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:48.724280 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:48.724480 15919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:57:48.734171 15919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:48.735476 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:48.736846 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:48.749905 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:48.749931 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:48.750602 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:48.750895 15919 server_base.cc:1034] running on GCE node
I20260929 00:57:48.751477 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:48.752413 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:48.753659 15919 hybrid_clock.cc:648] HybridClock initialized: now 1790643468753607 us; error 45 us; skew 500 ppm
I20260929 00:57:48.753961 15919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:48 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:48.757081 15919 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:57:48.758632 15919 webserver.cc:460] Webserver started at http://127.0.97.1:46345/ using document root <none> and password file <none>
I20260929 00:57:48.759325 15919 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:48.759481 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:48.759809 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:48.762739 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "bb28377a51814da190236b3e73d3b1b1"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "3c8a55c6972c8b2cd6f33c147bf18bda"
server_key_iv: "c93c4b3983ce0e008c30e6bf18027a02"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.763505 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bb28377a51814da190236b3e73d3b1b1"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "3c8a55c6972c8b2cd6f33c147bf18bda"
server_key_iv: "c93c4b3983ce0e008c30e6bf18027a02"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.768606 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 00:57:48.772022 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:48.773463 15919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:57:48.773653 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "bb28377a51814da190236b3e73d3b1b1"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "3c8a55c6972c8b2cd6f33c147bf18bda"
server_key_iv: "c93c4b3983ce0e008c30e6bf18027a02"
server_key_version: "encryptionkey@0"
I20260929 00:57:48.773847 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:48.791046 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:48.795516 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:48.795746 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:48.796649 15919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:48.797861 15919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:48.797930 15919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:48.798013 15919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:48.798054 15919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:48.830605 15919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39451
I20260929 00:57:48.830669 16046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39451 every 8 connection(s)
I20260929 00:57:48.832249 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:57:48.841149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 15919
I20260929 00:57:48.841362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:57:48.841967 388 external_mini_cluster.cc:1428] Setting key 16a07fecbd06a106fcd9163e51dba1f0
Sep 29 00:57:48 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:48.852865 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:48.834979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42239 (local address 127.0.97.62:43469)
0929 00:57:48.835457 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:48.835461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:48.836441 (+ 980us) server_negotiation.cc:407] Connection header received
0929 00:57:48.838314 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:48.838319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:48.838405 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:48.838535 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:48.839994 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.840786 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.842475 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.843031 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.843166 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:48.846972 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:48.846978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:48.846981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:48.847016 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:48.849634 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:48.850464 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:48.850469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:48.850472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:48.850614 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:48.851039 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:48.851045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:48.851049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:48.851314 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:48.851482 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:48.852343 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 00:57:48.852595 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":263,"threads_started":1}
I20260929 00:57:48.854352 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:48.835545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:42239)
0929 00:57:48.836245 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:48.836307 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:48.838037 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:48.838767 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:48.838775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:48.839267 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:48.839814 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:48.839830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.840945 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.840957 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:48.842316 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:48.842327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:48.843275 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:48.843283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:48.843523 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:48.844280 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:48.844308 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:48.846729 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:48.849782 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:48.849791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:48.849808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:48.850340 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:48.850728 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:48.850733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:48.850737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:48.850928 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:48.851446 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:48.851454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:48.851926 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 00:57:48.852240 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 00:57:48.852866 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":264,"threads_started":1}
I20260929 00:57:48.855883 16047 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:57:48.856200 16047 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:48.857026 16047 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:57:48.859082 15858 ts_manager.cc:194] Registered new tserver with Master: bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451)
I20260929 00:57:48.860616 15858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42239
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:57:48.895879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:49.216362 16053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:49.216713 16053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:49.216895 16053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:49.224212 16053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:49.224351 16053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:49.224563 16053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:57:49.239923 16053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:49.241591 16053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:49.243245 16053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:49.257483 16061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:49.257457 16059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:49.257485 16058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:49.263412 16053 server_base.cc:1034] running on GCE node
I20260929 00:57:49.264081 16053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:49.265141 16053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:49.266400 16053 hybrid_clock.cc:648] HybridClock initialized: now 1790643469266348 us; error 34 us; skew 500 ppm
I20260929 00:57:49.266758 16053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:49 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:49.270110 16053 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:57:49.271644 16053 webserver.cc:460] Webserver started at http://127.0.97.2:32861/ using document root <none> and password file <none>
I20260929 00:57:49.272377 16053 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:49.272572 16053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:49.272936 16053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:49.275907 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "f8d31e58a4941a3b833d88aec1b9782a"
server_key_iv: "a13aeb9ea38558ed805dfd19ea14dc3d"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.276712 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "f8d31e58a4941a3b833d88aec1b9782a"
server_key_iv: "a13aeb9ea38558ed805dfd19ea14dc3d"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.281745 16053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:57:49.285209 16068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.286468 16053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:57:49.286639 16053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "f8d31e58a4941a3b833d88aec1b9782a"
server_key_iv: "a13aeb9ea38558ed805dfd19ea14dc3d"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.286862 16053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:49.311331 16053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:49.315234 16053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:49.315529 16053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:49.316490 16053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:49.317785 16053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:49.317893 16053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.318040 16053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:49.318112 16053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.350262 16053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33207
I20260929 00:57:49.350303 16180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33207 every 8 connection(s)
I20260929 00:57:49.351645 16053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:57:49.358055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16053
I20260929 00:57:49.358275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:57:49.358695 388 external_mini_cluster.cc:1428] Setting key d2f934728ebe3011a917a284eb935200
Sep 29 00:57:49 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:49.373221 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.354283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42623 (local address 127.0.97.62:43469)
0929 00:57:49.354741 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.354748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.355808 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 00:57:49.357703 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.357710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.357800 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.357936 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:49.359606 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.360474 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.362061 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.362830 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.362982 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.366604 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:49.366613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:49.366620 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:49.366666 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:49.369843 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.370740 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.370744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.370746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.370867 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.371322 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.371325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.371327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.371567 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:49.371674 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.372654 (+ 980us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.372822 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260929 00:57:49.374651 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.354826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:42623)
0929 00:57:49.355654 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:49.355693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:49.357425 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:49.358118 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:49.358125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:49.358733 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:49.359458 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.359479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.360658 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.360663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.361909 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.361921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.363029 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.363033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.363223 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.363921 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:49.363942 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:49.366365 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:49.369965 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:49.369977 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:49.369997 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:49.370630 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:49.370969 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:49.370977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:49.370983 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:49.371221 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:49.371701 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:49.371711 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:49.372156 (+ 445us) client_negotiation.cc:769] Sending connection context
0929 00:57:49.372483 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 00:57:49.373147 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":321,"threads_started":1}
I20260929 00:57:49.376191 16181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:57:49.376549 16181 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:49.377394 16181 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:57:49.378842 15858 ts_manager.cc:194] Registered new tserver with Master: 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207)
I20260929 00:57:49.379655 15858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42623
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:57:49.412011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:57:49.736042 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:57:49.736322 16187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:57:49.736416 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:57:49.743649 16187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:57:49.743750 16187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:57:49.743870 16187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:57:49.753682 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:57:49.754930 16187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:57:49.756194 16187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:57:49.769443 16192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:49.769483 16195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:57:49.769410 16193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:57:49.770375 16187 server_base.cc:1034] running on GCE node
I20260929 00:57:49.770964 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:57:49.771845 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:57:49.773065 16187 hybrid_clock.cc:648] HybridClock initialized: now 1790643469773024 us; error 56 us; skew 500 ppm
I20260929 00:57:49.773379 16187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:57:49 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:57:49.776363 16187 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:57:49.777849 16187 webserver.cc:460] Webserver started at http://127.0.97.3:41391/ using document root <none> and password file <none>
I20260929 00:57:49.778573 16187 fs_manager.cc:359] Metadata directory not provided
I20260929 00:57:49.778714 16187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:57:49.779018 16187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:57:49.781845 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "fe25e771666c4a6f83b01709d78c1a66"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "9e3721050d99c783237783bbc72e935a"
server_key_iv: "8666db5eb87e25ca59506092254c8ffb"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.782580 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fe25e771666c4a6f83b01709d78c1a66"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "9e3721050d99c783237783bbc72e935a"
server_key_iv: "8666db5eb87e25ca59506092254c8ffb"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.787413 16187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 00:57:49.790814 16202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.792042 16187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:57:49.792202 16187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "fe25e771666c4a6f83b01709d78c1a66"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "9e3721050d99c783237783bbc72e935a"
server_key_iv: "8666db5eb87e25ca59506092254c8ffb"
server_key_version: "encryptionkey@0"
I20260929 00:57:49.792390 16187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:57:49.806227 16187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:57:49.810002 16187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:57:49.810285 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:57:49.811193 16187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:57:49.812439 16187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:57:49.812561 16187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.812714 16187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:57:49.812788 16187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:57:49.845168 16187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35335
I20260929 00:57:49.845208 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35335 every 8 connection(s)
I20260929 00:57:49.846485 16187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:57:49.854745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16187
I20260929 00:57:49.854945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:57:49.855433 388 external_mini_cluster.cc:1428] Setting key b41d0b2f27b3eda9095da991ed04b970
Sep 29 00:57:49 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:57:49.863428 16047 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:57:49.866050 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.849033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49283 (local address 127.0.97.62:43469)
0929 00:57:49.849222 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.849226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.850358 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 00:57:49.852225 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.852231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.852337 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.852451 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:49.853921 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.854847 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.856649 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.857194 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.857333 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.860756 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:49.860761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:49.860764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:49.860795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:49.863060 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.863789 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.863793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.863795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.863955 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.864441 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.864444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.864446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.864824 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:49.864978 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.865655 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.865826 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:57:49.867015 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.849515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:49283)
0929 00:57:49.850206 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:49.850248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:49.851933 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:49.852705 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:49.852713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:49.853216 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:57:49.853752 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.853768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.854987 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.854991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.856467 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.856479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.857442 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.857450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.857665 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.858399 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:57:49.858421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:57:49.860572 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:57:49.863180 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:49.863186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:49.863198 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:49.863661 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:49.864087 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:57:49.864092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:57:49.864095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:57:49.864330 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:57:49.864967 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:57:49.864974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:57:49.865317 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 00:57:49.865545 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 00:57:49.865912 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":288,"threads_started":1}
I20260929 00:57:49.868386 16315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:57:49.868691 16315 heartbeater.cc:464] Registering TS with master...
I20260929 00:57:49.869580 16315 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:57:49.871017 15858 ts_manager.cc:194] Registered new tserver with Master: fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335)
I20260929 00:57:49.871789 15858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49283
I20260929 00:57:49.882452 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:57:49.898838 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.886689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.97.62:43469)
0929 00:57:49.886840 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.886844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.887338 (+ 494us) server_negotiation.cc:407] Connection header received
0929 00:57:49.887562 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.887566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.887623 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.887699 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:49.889127 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.889904 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.891280 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.891656 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.891744 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.894198 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:49.894202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:49.894205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:49.894232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:49.896540 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.897245 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.897249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.897251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.897366 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:49.897690 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:49.897693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:49.897695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:49.897992 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:49.898169 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.898420 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.898598 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:57:49.902681 15858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:57:49.909449 15858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:57:49.934028 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.921466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:42998)
0929 00:57:49.922232 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:49.922274 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:49.922427 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:49.924013 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:49.924023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:49.924062 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:49.924898 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.924918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.927481 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.927489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.930425 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.930445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.932812 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.932820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.933110 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.933184 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 00:57:49.933299 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 00:57:49.933542 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":388,"threads_started":1}
I20260929 00:57:49.935734 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.921639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.0.97.1:39451)
0929 00:57:49.923336 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.923345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.923372 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:57:49.923488 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.923498 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.923768 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.925169 (+ 1401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:49.925449 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.927303 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.930687 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.932558 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.934745 (+ 2187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.935038 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.935115 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.935316 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":1168,"threads_started":1}
I20260929 00:57:49.940470 15982 tablet_service.cc:1467] Processing CreateTablet for tablet fa02a36db01647c9845f231d7905f41e (DEFAULT_TABLE table=test-table [id=53759652ba6e4ca8a153ec7de560b896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:49.941991 15982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa02a36db01647c9845f231d7905f41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:49.944101 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.922607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35335 (local address 127.0.0.1:47210)
0929 00:57:49.934393 (+ 11786us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:49.934414 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:49.934555 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:49.937276 (+ 2721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:49.937282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:49.937310 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:49.937470 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.937481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.939464 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.939470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.941501 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.941516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.943225 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.943231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.943432 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.943465 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:57:49.943536 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 00:57:49.943674 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11656,"mutex_wait_us":1,"thread_start_us":14229,"threads_started":1}
I20260929 00:57:49.944141 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.923150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33207 (local address 127.0.0.1:40372)
0929 00:57:49.935523 (+ 12373us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:49.935541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:49.935650 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:49.937498 (+ 1848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:49.937504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:49.937530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:49.937860 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.937873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.939648 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.939653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.941684 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:49.941693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.943339 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.943346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:49.943538 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.943574 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:57:49.943656 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 00:57:49.943794 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12248,"thread_start_us":12120,"threads_started":1}
I20260929 00:57:49.944329 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.923366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.97.2:33207)
0929 00:57:49.936802 (+ 13436us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.936812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.936845 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:57:49.936943 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.936953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.937216 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.937418 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:49.938040 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.939503 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.941822 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.942840 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.942984 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.943183 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.943829 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.944006 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13268,"thread_start_us":13148,"threads_started":1}
I20260929 00:57:49.945300 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:49.923216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.0.97.3:35335)
0929 00:57:49.936546 (+ 13330us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:49.936559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:49.936601 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:57:49.936723 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:49.936734 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:49.937001 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:49.937217 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:49.937654 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.939297 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.941660 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:49.943045 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:49.944700 (+ 1655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:49.944857 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:49.944909 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 00:57:49.945013 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13107,"thread_start_us":12939,"threads_started":1}
I20260929 00:57:49.947993 16116 tablet_service.cc:1467] Processing CreateTablet for tablet fa02a36db01647c9845f231d7905f41e (DEFAULT_TABLE table=test-table [id=53759652ba6e4ca8a153ec7de560b896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:49.948784 16250 tablet_service.cc:1467] Processing CreateTablet for tablet fa02a36db01647c9845f231d7905f41e (DEFAULT_TABLE table=test-table [id=53759652ba6e4ca8a153ec7de560b896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:57:49.949465 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa02a36db01647c9845f231d7905f41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:49.950131 16250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa02a36db01647c9845f231d7905f41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:57:49.955219 16333 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Bootstrap starting.
I20260929 00:57:49.958433 16333 tablet_bootstrap.cc:654] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:49.959381 16333 log.cc:824] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:49.962332 16333 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: No bootstrap required, opened a new log
I20260929 00:57:49.962611 16333 ts_tablet_manager.cc:1397] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 00:57:49.965198 16336 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Bootstrap starting.
I20260929 00:57:49.966949 16335 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Bootstrap starting.
I20260929 00:57:49.968726 16336 tablet_bootstrap.cc:654] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:49.969916 16336 log.cc:824] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:49.970515 16335 tablet_bootstrap.cc:654] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Neither blocks nor log segments found. Creating new log.
I20260929 00:57:49.971639 16335 log.cc:824] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Log is configured to *not* fsync() on all Append() calls
I20260929 00:57:49.973037 16336 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: No bootstrap required, opened a new log
I20260929 00:57:49.972857 16333 raft_consensus.cc:348] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.973278 16336 ts_tablet_manager.cc:1397] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 00:57:49.973333 16333 raft_consensus.cc:374] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:49.973433 16333 raft_consensus.cc:729] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb28377a51814da190236b3e73d3b1b1, State: Initialized, Role: FOLLOWER
I20260929 00:57:49.974079 16333 consensus_queue.cc:260] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.974745 16335 tablet_bootstrap.cc:492] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: No bootstrap required, opened a new log
I20260929 00:57:49.975826 16333 ts_tablet_manager.cc:1428] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260929 00:57:49.976111 16335 ts_tablet_manager.cc:1397] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 00:57:49.982553 16336 raft_consensus.cc:348] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.982887 16336 raft_consensus.cc:374] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:49.982990 16336 raft_consensus.cc:729] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3d01d4a26c43efbaa94fcc79f2fca0, State: Initialized, Role: FOLLOWER
I20260929 00:57:49.983596 16336 consensus_queue.cc:260] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.984161 16335 raft_consensus.cc:348] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.984542 16335 raft_consensus.cc:374] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:57:49.984640 16335 raft_consensus.cc:729] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe25e771666c4a6f83b01709d78c1a66, State: Initialized, Role: FOLLOWER
I20260929 00:57:49.984655 16181 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:57:49.985216 16336 ts_tablet_manager.cc:1428] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260929 00:57:49.985155 16335 consensus_queue.cc:260] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:49.986121 16315 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:57:49.986701 16335 ts_tablet_manager.cc:1428] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
W20260929 00:57:50.084765 16048 tablet.cc:2369] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:50.086143 16340 raft_consensus.cc:482] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:57:50.086405 16340 raft_consensus.cc:504] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:50.088090 16340 leader_election.cc:288] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451), fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335)
I20260929 00:57:50.095393 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.088584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52529 (local address 127.0.97.1:39451)
0929 00:57:50.088818 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.088827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.089298 (+ 471us) server_negotiation.cc:407] Connection header received
0929 00:57:50.089562 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.089570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.089659 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.089761 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:50.090460 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.091429 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.093406 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.094409 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.094520 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.094592 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.094984 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.095117 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:57:50.095446 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.088453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.97.2:52529)
0929 00:57:50.089192 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:50.089212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:50.089418 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:50.089796 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:50.089802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:50.089847 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:50.090296 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:50.090308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.091574 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.091580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:50.093250 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:50.093261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.094616 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.094623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:50.094792 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.094828 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:57:50.094935 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 00:57:50.095063 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":419,"threads_started":1}
I20260929 00:57:50.095831 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.089565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37949 (local address 127.0.97.3:35335)
0929 00:57:50.089990 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.089997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.090104 (+ 107us) server_negotiation.cc:407] Connection header received
0929 00:57:50.090265 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.090271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.090349 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.090438 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:57:50.091072 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.091989 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.093965 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.094864 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.094986 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.095076 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.095450 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.095579 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260929 00:57:50.095871 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.089194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35335 (local address 127.0.97.2:37949)
0929 00:57:50.090017 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 00:57:50.090030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:57:50.090125 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:57:50.090482 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:57:50.090488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:57:50.090514 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:57:50.090923 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:50.090937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.092153 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.092157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:50.093810 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:57:50.093826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.095091 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.095097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:57:50.095285 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.095316 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:57:50.095420 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 00:57:50.095572 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":483,"threads_started":1}
I20260929 00:57:50.096916 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa02a36db01647c9845f231d7905f41e" candidate_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe25e771666c4a6f83b01709d78c1a66" is_pre_election: true
I20260929 00:57:50.097018 16002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa02a36db01647c9845f231d7905f41e" candidate_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb28377a51814da190236b3e73d3b1b1" is_pre_election: true
I20260929 00:57:50.097447 16270 raft_consensus.cc:2454] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3d01d4a26c43efbaa94fcc79f2fca0 in term 0.
I20260929 00:57:50.097532 16002 raft_consensus.cc:2454] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3d01d4a26c43efbaa94fcc79f2fca0 in term 0.
W20260929 00:57:50.098026 16316 tablet.cc:2369] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:50.098093 16072 leader_election.cc:302] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3d01d4a26c43efbaa94fcc79f2fca0, fe25e771666c4a6f83b01709d78c1a66; no voters:
I20260929 00:57:50.098497 16340 raft_consensus.cc:2781] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:57:50.098654 16340 raft_consensus.cc:482] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:57:50.098759 16340 raft_consensus.cc:3037] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:50.100981 16340 raft_consensus.cc:504] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:50.101575 16340 leader_election.cc:288] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [CANDIDATE]: Term 1 election: Requested vote from peers bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451), fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335)
I20260929 00:57:50.101977 16002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa02a36db01647c9845f231d7905f41e" candidate_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb28377a51814da190236b3e73d3b1b1"
I20260929 00:57:50.102015 16270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa02a36db01647c9845f231d7905f41e" candidate_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe25e771666c4a6f83b01709d78c1a66"
I20260929 00:57:50.102188 16002 raft_consensus.cc:3037] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:57:50.102192 16270 raft_consensus.cc:3037] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Advancing to term 1
W20260929 00:57:50.103536 16182 tablet.cc:2369] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:57:50.104394 16270 raft_consensus.cc:2454] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3d01d4a26c43efbaa94fcc79f2fca0 in term 1.
I20260929 00:57:50.104772 16072 leader_election.cc:302] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3d01d4a26c43efbaa94fcc79f2fca0, fe25e771666c4a6f83b01709d78c1a66; no voters:
I20260929 00:57:50.104871 16002 raft_consensus.cc:2454] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3d01d4a26c43efbaa94fcc79f2fca0 in term 1.
I20260929 00:57:50.105077 16340 raft_consensus.cc:2781] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:57:50.105590 16340 raft_consensus.cc:686] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 LEADER]: Becoming Leader. State: Replica: 6d3d01d4a26c43efbaa94fcc79f2fca0, State: Running, Role: LEADER
I20260929 00:57:50.106035 16340 consensus_queue.cc:237] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:57:50.110711 15858 catalog_manager.cc:5495] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } health_report { overall_health: HEALTHY } } }
I20260929 00:57:50.134105 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.129139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.0.97.2:33207)
0929 00:57:50.129355 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.129360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.129379 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:57:50.129576 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.129580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.129656 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.129780 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:57:50.130093 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.130817 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.132111 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.132642 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.132740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.133137 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:57:50.133647 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:57:50.133752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.133837 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.133899 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:57:50.142354 16270 raft_consensus.cc:1261] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Refusing update from remote peer 6d3d01d4a26c43efbaa94fcc79f2fca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:50.142372 16002 raft_consensus.cc:1261] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Refusing update from remote peer 6d3d01d4a26c43efbaa94fcc79f2fca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:57:50.143301 16344 consensus_queue.cc:1035] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:50.143638 16340 consensus_queue.cc:1035] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:57:50.172724 15858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:50.173017 15858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:57:50.177783 15858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Sending DeleteTablet for 3 replicas of tablet fa02a36db01647c9845f231d7905f41e
I20260929 00:57:50.178928 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet fa02a36db01647c9845f231d7905f41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40372
I20260929 00:57:50.179700 15982 tablet_service.cc:1514] Processing DeleteTablet for tablet fa02a36db01647c9845f231d7905f41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42998
I20260929 00:57:50.179903 16357 tablet_replica.cc:330] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: stopping tablet replica
I20260929 00:57:50.180344 16357 raft_consensus.cc:2229] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:57:50.180452 16250 tablet_service.cc:1514] Processing DeleteTablet for tablet fa02a36db01647c9845f231d7905f41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47210
I20260929 00:57:50.180605 16358 tablet_replica.cc:330] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: stopping tablet replica
I20260929 00:57:50.180989 16358 raft_consensus.cc:2229] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:50.181041 16357 raft_consensus.cc:2258] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:50.181499 16358 raft_consensus.cc:2258] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:50.182739 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet fa02a36db01647c9845f231d7905f41e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:57:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40372
I20260929 00:57:50.183009 16357 ts_tablet_manager.cc:1905] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:50.183362 16358 ts_tablet_manager.cc:1905] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:57:50.183763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15919
I20260929 00:57:50.190670 16358 ts_tablet_manager.cc:1918] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:50.190863 16358 log.cc:1196] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/fa02a36db01647c9845f231d7905f41e
I20260929 00:57:50.191470 16358 ts_tablet_manager.cc:1939] T fa02a36db01647c9845f231d7905f41e P bb28377a51814da190236b3e73d3b1b1: Deleting consensus metadata
I20260929 00:57:50.191803 16357 ts_tablet_manager.cc:1918] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:50.192040 16357 log.cc:1196] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/fa02a36db01647c9845f231d7905f41e
I20260929 00:57:50.192672 16357 ts_tablet_manager.cc:1939] T fa02a36db01647c9845f231d7905f41e P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting consensus metadata
I20260929 00:57:50.193423 15845 catalog_manager.cc:4841] TS bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451): tablet fa02a36db01647c9845f231d7905f41e (table test-table [id=53759652ba6e4ca8a153ec7de560b896]) successfully deleted
I20260929 00:57:50.194566 15844 catalog_manager.cc:4841] TS 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207): tablet fa02a36db01647c9845f231d7905f41e (table test-table [id=53759652ba6e4ca8a153ec7de560b896]) successfully deleted
W20260929 00:57:50.195153 15844 catalog_manager.cc:4808] TS 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207): delete failed for tablet fa02a36db01647c9845f231d7905f41e because the tablet was not found. No further retry: Not found: Tablet not found: fa02a36db01647c9845f231d7905f41e
I20260929 00:57:50.200162 16359 tablet_replica.cc:330] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: stopping tablet replica
I20260929 00:57:50.200666 16359 raft_consensus.cc:2229] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:57:50.201246 16359 raft_consensus.cc:2258] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:57:50.204782 16359 ts_tablet_manager.cc:1905] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 00:57:50 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:57:50.212087 16359 ts_tablet_manager.cc:1918] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:57:50.212273 16359 log.cc:1196] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/fa02a36db01647c9845f231d7905f41e
I20260929 00:57:50.213187 16359 ts_tablet_manager.cc:1939] T fa02a36db01647c9845f231d7905f41e P fe25e771666c4a6f83b01709d78c1a66: Deleting consensus metadata
I20260929 00:57:50.215186 15845 catalog_manager.cc:4841] TS fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335): tablet fa02a36db01647c9845f231d7905f41e (table test-table [id=53759652ba6e4ca8a153ec7de560b896]) successfully deleted
I20260929 00:57:50.216068 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.184871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.0.97.1:39451)
0929 00:57:50.185033 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.185043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.199410 (+ 14367us) server_negotiation.cc:407] Connection header received
0929 00:57:50.199633 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.199639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.199728 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.199844 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:50.201993 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.203115 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.204547 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.204956 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.205105 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.209671 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:50.209687 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:50.209702 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:50.209735 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:50.212909 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.213715 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.213725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.213728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.213865 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.214273 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.214282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.214284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.214659 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:50.214866 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.215488 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.215715 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:57:50.216724 16042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:50.307438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15919
I20260929 00:57:50.325064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16053
Sep 29 00:57:50 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:57:50.342111 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.326008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.97.2:33207)
0929 00:57:50.326218 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.326226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.326247 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:57:50.326320 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.326326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.326423 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.326531 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:50.327721 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.328987 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.330303 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.330987 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.331109 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.335171 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:50.335188 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:50.335208 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:50.335261 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:50.339170 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.339885 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.339895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.339899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.340089 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.340423 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.340430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.340433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.341016 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:50.341338 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.341481 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.341704 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:57:50.342941 16176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:50.454691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16053
I20260929 00:57:50.473647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16187
Sep 29 00:57:50 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:57:50.489130 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:57:50.474404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.0.97.3:35335)
0929 00:57:50.474574 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:57:50.474579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:57:50.474627 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:57:50.474776 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:57:50.474781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:57:50.474845 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:57:50.474927 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:57:50.476144 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.476963 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.478274 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:57:50.478643 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:57:50.478730 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:57:50.483185 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:57:50.483199 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:57:50.483217 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:57:50.483255 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:57:50.486141 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.486858 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.486865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.486868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.487028 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:57:50.487422 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:57:50.487427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:57:50.487431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:57:50.487870 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:57:50.488166 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 00:57:50.488485 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 00:57:50.488745 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:57:50.489828 16310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:50.579730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16187
I20260929 00:57:50.596673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 15826
I20260929 00:57:50.597792 15888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:57:50.707762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 15826
I20260929 00:58:10.729197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--webserver_interface=127.0.97.62
--webserver_port=33859
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:11.051788 16365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:11.052109 16365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:11.052274 16365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:11.059216 16365 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:58:11.059342 16365 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:58:11.059443 16365 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:58:11.059537 16365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:11.059614 16365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:11.059713 16365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:11.059780 16365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:11.059862 16365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:11.069259 16365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:11.070585 16365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:11.071862 16365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:11.094617 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:11.094602 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:11.094651 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:11.098062 16365 server_base.cc:1034] running on GCE node
I20260929 00:58:11.098841 16365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:11.100333 16365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:11.101586 16365 hybrid_clock.cc:648] HybridClock initialized: now 1790643491101519 us; error 42 us; skew 500 ppm
I20260929 00:58:11.101956 16365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:11 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:11.105552 16365 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:58:11.107052 16365 webserver.cc:460] Webserver started at http://127.0.97.62:33859/ using document root <none> and password file <none>
I20260929 00:58:11.107960 16365 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:11.108104 16365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:11.114316 16365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:58:11.117172 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:11.118386 16365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:11.118572 16365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0c90c3c984b1424181f61293beb75439"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "0bc1fcacc668aa2b3ff41ad81a2d0e59"
server_key_iv: "2230176c891faa60b3b77e93ca7b3b0d"
server_key_version: "encryptionkey@0"
I20260929 00:58:11.119586 16365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:11.142213 16365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:11.146973 16365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:11.147277 16365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:11.166534 16365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43469
I20260929 00:58:11.166556 16432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43469 every 8 connection(s)
I20260929 00:58:11.168107 16365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:11.171564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16365
I20260929 00:58:11.172359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39451
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=46345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:58:11.175429 16433 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:58:11.178836 16433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap starting.
I20260929 00:58:11.209834 16433 log.cc:824] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:11.225162 16433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:58:11.225908 16433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap complete.
I20260929 00:58:11.235948 16433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:11.236917 16433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Initialized, Role: FOLLOWER
I20260929 00:58:11.237563 16433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:11.237804 16433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:11.237924 16433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:11.238121 16433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 1 FOLLOWER]: Advancing to term 2
I20260929 00:58:11.242057 16433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:11.242576 16433 leader_election.cc:302] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c90c3c984b1424181f61293beb75439; no voters:
I20260929 00:58:11.243268 16433 leader_election.cc:288] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 00:58:11.243402 16437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 00:58:11.244639 16437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 LEADER]: Becoming Leader. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Running, Role: LEADER
I20260929 00:58:11.245252 16437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:11.245581 16433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:11.247728 16439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c90c3c984b1424181f61293beb75439. Latest consensus state: current_term: 2 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:58:11.248126 16439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:11.248683 16438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:58:11.248999 16438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:11.251235 16443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:11.260051 16443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:11.261492 16443 catalog_manager.cc:1259] Loaded cluster ID: 4aad156ec0b2404ab2fc5bb48a29609a
I20260929 00:58:11.261648 16443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:11.265626 16443 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:11.267155 16443 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Loaded TSK: 0
I20260929 00:58:11.301093 16443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Generated new TSK 1
I20260929 00:58:11.301507 16443 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 00:58:11.535596 16435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:11.535924 16435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:11.536087 16435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:11.543401 16435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:11.543573 16435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:11.543743 16435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:58:11.554193 16435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46345
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:11.555503 16435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:11.556900 16435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:11.571611 16461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:11.571856 16460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:11.572824 16463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:11.573065 16435 server_base.cc:1034] running on GCE node
I20260929 00:58:11.573752 16435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:11.574798 16435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:11.576023 16435 hybrid_clock.cc:648] HybridClock initialized: now 1790643491575959 us; error 59 us; skew 500 ppm
I20260929 00:58:11.576347 16435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:11 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:11.579681 16435 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:58:11.581387 16435 webserver.cc:460] Webserver started at http://127.0.97.1:46345/ using document root <none> and password file <none>
I20260929 00:58:11.582100 16435 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:11.582252 16435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:11.587816 16435 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:58:11.590646 16470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:11.592062 16435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:58:11.592229 16435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "bb28377a51814da190236b3e73d3b1b1"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "3c8a55c6972c8b2cd6f33c147bf18bda"
server_key_iv: "c93c4b3983ce0e008c30e6bf18027a02"
server_key_version: "encryptionkey@0"
I20260929 00:58:11.593266 16435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:11.606417 16435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:11.610739 16435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:11.611032 16435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:11.612093 16435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:11.613494 16435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:11.613591 16435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:11.613730 16435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:11.613824 16435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:11.646706 16435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39451
I20260929 00:58:11.646791 16582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39451 every 8 connection(s)
I20260929 00:58:11.648192 16435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:58:11.649885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16435
I20260929 00:58:11.650691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33207
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=32861
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:58:11 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:11.674520 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:11.651595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:33281)
0929 00:58:11.652490 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:11.652575 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:11.654361 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:11.655963 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:11.655970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:11.656483 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:11.657102 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:11.657120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:11.658974 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:11.658982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:11.660329 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:11.660342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:11.661872 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:11.661877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:11.662066 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:11.662786 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:11.662809 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:11.665262 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:11.670286 (+ 5024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:11.670293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:11.670306 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:11.670748 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:11.671273 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:11.671276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:11.671278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:11.671455 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:11.672483 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:11.672490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:11.672848 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 00:58:11.673078 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 00:58:11.673489 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":404,"threads_started":1}
I20260929 00:58:11.675235 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:11.651199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33281 (local address 127.0.97.62:43469)
0929 00:58:11.651860 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:11.651879 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:11.652707 (+ 828us) server_negotiation.cc:407] Connection header received
0929 00:58:11.654954 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:11.654975 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:11.655449 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:11.655846 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:11.657314 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:11.658767 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:11.660519 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:11.661657 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:11.661846 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:11.665583 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:11.665598 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:11.665623 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:11.665683 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:11.670127 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:11.670907 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:11.670920 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:11.670929 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:11.671159 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:11.671584 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:11.671592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:11.671598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:11.672333 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:11.672595 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:11.673383 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 00:58:11.673884 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260929 00:58:11.677983 16583 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:58:11.678320 16583 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:11.679170 16583 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:58:11.681638 16398 ts_manager.cc:194] Registered new tserver with Master: bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451)
I20260929 00:58:11.683772 16398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33281
W20260929 00:58:11.980731 16587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:11.981041 16587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:11.981199 16587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:11.988408 16587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:11.988579 16587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:11.988758 16587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:58:11.998466 16587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32861
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:11.999668 16587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:12.000982 16587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:12.014063 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:12.014086 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:12.014213 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:12.014711 16587 server_base.cc:1034] running on GCE node
I20260929 00:58:12.015447 16587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:12.016350 16587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:12.017544 16587 hybrid_clock.cc:648] HybridClock initialized: now 1790643492017515 us; error 58 us; skew 500 ppm
I20260929 00:58:12.017887 16587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:12 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:12.020947 16587 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:58:12.022274 16587 webserver.cc:460] Webserver started at http://127.0.97.2:32861/ using document root <none> and password file <none>
I20260929 00:58:12.022908 16587 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:12.023027 16587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:12.028236 16587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:58:12.030773 16602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.032367 16587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:58:12.032572 16587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "f8d31e58a4941a3b833d88aec1b9782a"
server_key_iv: "a13aeb9ea38558ed805dfd19ea14dc3d"
server_key_version: "encryptionkey@0"
I20260929 00:58:12.034062 16587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:12.051710 16587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:12.055666 16587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:12.055934 16587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:12.056917 16587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:12.058169 16587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:12.058279 16587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.058394 16587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:12.058485 16587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.090338 16587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33207
I20260929 00:58:12.090374 16714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33207 every 8 connection(s)
I20260929 00:58:12.091738 16587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:58:12.096822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16587
I20260929 00:58:12.097549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35335
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=41391
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 00:58:12 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:12.112200 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.094223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38985 (local address 127.0.97.62:43469)
0929 00:58:12.094413 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.094418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.095541 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 00:58:12.097523 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.097531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.097627 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.097745 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:12.099341 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.100466 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.102145 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.102901 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.103051 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.106388 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:12.106393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:12.106396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:12.106428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:12.109065 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.109877 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.109887 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.109889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.110007 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.110416 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.110422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.110425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.110804 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:12.110961 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.111617 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.111859 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:58:12.113257 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.094716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:38985)
0929 00:58:12.095392 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.095432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.097251 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.098085 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.098095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.098613 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:12.099167 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.099183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.100679 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.100687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.102002 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.102014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.103104 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.103108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.103296 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.103973 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:12.103995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:12.106170 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:12.109209 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:12.109218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:12.109235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:12.109755 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:12.110113 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:12.110119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:12.110122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:12.110308 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:12.110919 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:12.110927 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:12.111287 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.111543 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.112080 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":285,"threads_started":1}
I20260929 00:58:12.114763 16715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:58:12.115127 16715 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:12.116108 16715 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:58:12.117718 16398 ts_manager.cc:194] Registered new tserver with Master: 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207)
I20260929 00:58:12.118547 16398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38985
W20260929 00:58:12.426156 16718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:12.426426 16718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:12.426614 16718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:12.433614 16718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:12.433710 16718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:12.433872 16718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:58:12.443348 16718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41391
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:12.444620 16718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:12.445909 16718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:12.458607 16723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:12.458647 16726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:12.458693 16724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:12.459293 16718 server_base.cc:1034] running on GCE node
I20260929 00:58:12.459968 16718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:12.460937 16718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:12.462132 16718 hybrid_clock.cc:648] HybridClock initialized: now 1790643492462089 us; error 32 us; skew 500 ppm
I20260929 00:58:12.462463 16718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:12 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:12.465581 16718 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:58:12.467031 16718 webserver.cc:460] Webserver started at http://127.0.97.3:41391/ using document root <none> and password file <none>
I20260929 00:58:12.467694 16718 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:12.467842 16718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:12.473271 16718 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:58:12.475967 16733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.477505 16718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:58:12.477716 16718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "fe25e771666c4a6f83b01709d78c1a66"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "9e3721050d99c783237783bbc72e935a"
server_key_iv: "8666db5eb87e25ca59506092254c8ffb"
server_key_version: "encryptionkey@0"
I20260929 00:58:12.478750 16718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:12.507973 16718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:12.512269 16718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:12.512605 16718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:12.513576 16718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:12.514894 16718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:12.515022 16718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.515152 16718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:12.515237 16718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:12.548285 16718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35335
I20260929 00:58:12.548414 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35335 every 8 connection(s)
I20260929 00:58:12.549885 16718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:58:12.551604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16718
Sep 29 00:58:12 dist-test-slave-pzsx krb5kdc[15813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:12.569348 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.552643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.0.97.62:43469)
0929 00:58:12.552855 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.552861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.553554 (+ 693us) server_negotiation.cc:407] Connection header received
0929 00:58:12.553906 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.553915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.554029 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.554122 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:12.556886 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.557808 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.559484 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.560075 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.560202 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.563354 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:12.563360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:12.563364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:12.563401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:12.566386 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.567091 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.567096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.567098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.567281 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.567662 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.567668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.567673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.568095 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:12.568376 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.568715 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.568942 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:58:12.573020 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.552814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51297 (local address 127.0.97.62:43469)
0929 00:58:12.556870 (+ 4056us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.556875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.556890 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:58:12.556930 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.556935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.557032 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.557153 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:12.558640 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.559675 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.561517 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.562230 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.562374 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.566553 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:12.566560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:12.566564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:12.566600 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:12.569841 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.570656 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.570677 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.570683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.570819 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.571307 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.571311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.571313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.571559 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:12.571691 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.572467 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.572709 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3927,"thread_start_us":3860,"threads_started":1}
I20260929 00:58:12.573921 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.553588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:51297)
0929 00:58:12.554413 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.554464 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.556155 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.557379 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.557388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.557900 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:12.558451 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.558469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.559821 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.559827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.561335 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.561348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.562474 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.562480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.562685 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.563365 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:12.563428 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:12.566297 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:12.569982 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:12.569992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:12.570015 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:12.570544 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:12.570938 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:12.570943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:12.570946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:12.571203 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:12.571695 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:12.571703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:12.572114 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.572353 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.572817 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
I20260929 00:58:12.575289 16846 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43469
I20260929 00:58:12.575640 16846 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:12.576371 16846 heartbeater.cc:511] Master 127.0.97.62:43469 requested a full tablet report, sending...
I20260929 00:58:12.577806 16398 ts_manager.cc:194] Registered new tserver with Master: fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335)
I20260929 00:58:12.578565 16398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51297
I20260929 00:58:12.584328 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:58:12.590427 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.585211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.97.62:43469)
0929 00:58:12.585358 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.585362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.585851 (+ 489us) server_negotiation.cc:407] Connection header received
0929 00:58:12.586031 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.586034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.586171 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.586287 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:58:12.586671 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.587391 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.588826 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.589230 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.589320 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.589759 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:58:12.589971 (+ 212us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 00:58:12.590229 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60878: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 00:58:12.590548 16849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60878: token expired
W20260929 00:58:12.591071 16319 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:43469)
I20260929 00:58:12.603066 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.591557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.0.97.62:43469)
0929 00:58:12.591702 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.591706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.591719 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:58:12.591754 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.591757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.591806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.591887 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:12.593227 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.593930 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.595194 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.595574 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.595663 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.598407 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:12.598412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:12.598415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:12.598444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:12.600934 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.601532 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.601536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.601538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.601690 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.601987 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.601990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.601992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.602273 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:12.602472 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.602677 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.602828 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:58:12.606527 16398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:58:12.614069 16398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:58:12.649474 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.632861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44012)
0929 00:58:12.640703 (+ 7842us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.640721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.640843 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.641539 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.641543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.641571 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:12.642046 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.642061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.644205 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.644210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.646257 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.646276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.648529 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.648537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.648806 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.648844 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.648936 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.649076 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7720,"mutex_wait_us":79,"thread_start_us":7540,"threads_started":1}
I20260929 00:58:12.649714 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.627072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.2:33207)
0929 00:58:12.639668 (+ 12596us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.639679 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.639719 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:58:12.640255 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.640268 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.640641 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.640868 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:12.641741 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.643151 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.646460 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.647946 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.648088 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.648299 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.649192 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.649394 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12390,"thread_start_us":12241,"threads_started":1}
I20260929 00:58:12.650182 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.626589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33207 (local address 127.0.0.1:54246)
0929 00:58:12.639063 (+ 12474us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.639087 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.640038 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.640875 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.640884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.640914 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:12.641500 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.641529 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.643303 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.643308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.646276 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.646291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.648464 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.648470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.648686 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.648788 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.648866 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.648971 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12311,"mutex_wait_us":211,"spinlock_wait_cycles":11904,"thread_start_us":13155,"threads_started":1}
I20260929 00:58:12.650202 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.626308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.0.97.3:35335)
0929 00:58:12.640356 (+ 14048us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.640366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.640405 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:58:12.640536 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.640546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.640823 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.641237 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:12.641870 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.643415 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.646677 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.647999 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.649195 (+ 1196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.649416 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.649490 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.649834 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13579,"thread_start_us":13411,"threads_started":1}
I20260929 00:58:12.650445 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.632918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.0.97.1:39451)
0929 00:58:12.638023 (+ 5105us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.638036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.640992 (+ 2956us) server_negotiation.cc:407] Connection header received
0929 00:58:12.641094 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.641101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.641316 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.641502 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:12.642235 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.644074 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.646472 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.647988 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.648102 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.648321 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.649936 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.650102 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4885,"thread_start_us":4718,"threads_started":1}
I20260929 00:58:12.651623 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.625866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35335 (local address 127.0.0.1:52818)
0929 00:58:12.632170 (+ 6304us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.632212 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.632374 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.641047 (+ 8673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.641051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.641092 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:12.641659 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.641669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.643552 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.643557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.646515 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.646530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.648234 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.648240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.648460 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.648544 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.648664 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.648884 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6073,"mutex_wait_us":177,"spinlock_wait_cycles":1792,"thread_start_us":5908,"threads_started":1}
I20260929 00:58:12.654525 16650 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec (DEFAULT_TABLE table=test-table [id=60b7f1fb573e4fc6a6eab4f1cd37c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:12.654547 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec (DEFAULT_TABLE table=test-table [id=60b7f1fb573e4fc6a6eab4f1cd37c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:12.655007 16781 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec (DEFAULT_TABLE table=test-table [id=60b7f1fb573e4fc6a6eab4f1cd37c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:12.655926 16650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c400c5144592adf08a4fe48ddbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:12.656039 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c400c5144592adf08a4fe48ddbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:12.656213 16781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c400c5144592adf08a4fe48ddbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:12.669088 16862 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Bootstrap starting.
I20260929 00:58:12.671422 16863 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Bootstrap starting.
I20260929 00:58:12.672196 16864 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Bootstrap starting.
I20260929 00:58:12.672349 16862 tablet_bootstrap.cc:654] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:12.673705 16862 log.cc:824] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:12.676064 16863 tablet_bootstrap.cc:654] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:12.677330 16863 log.cc:824] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:12.678072 16862 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: No bootstrap required, opened a new log
I20260929 00:58:12.678345 16862 ts_tablet_manager.cc:1397] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 00:58:12.678424 16864 tablet_bootstrap.cc:654] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:12.679416 16864 log.cc:824] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:12.680485 16863 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: No bootstrap required, opened a new log
I20260929 00:58:12.680809 16863 ts_tablet_manager.cc:1397] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 00:58:12.681931 16864 tablet_bootstrap.cc:492] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: No bootstrap required, opened a new log
I20260929 00:58:12.682240 16864 ts_tablet_manager.cc:1397] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 00:58:12.686995 16583 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:58:12.688863 16862 raft_consensus.cc:348] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.689313 16862 raft_consensus.cc:374] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:12.689437 16862 raft_consensus.cc:729] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3d01d4a26c43efbaa94fcc79f2fca0, State: Initialized, Role: FOLLOWER
I20260929 00:58:12.690181 16862 consensus_queue.cc:260] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.690332 16864 raft_consensus.cc:348] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.690768 16864 raft_consensus.cc:374] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:12.690863 16864 raft_consensus.cc:729] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe25e771666c4a6f83b01709d78c1a66, State: Initialized, Role: FOLLOWER
I20260929 00:58:12.691488 16864 consensus_queue.cc:260] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.692744 16846 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:58:12.693300 16864 ts_tablet_manager.cc:1428] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 00:58:12.694041 16715 heartbeater.cc:503] Master 127.0.97.62:43469 was elected leader, sending a full tablet report...
I20260929 00:58:12.694757 16862 ts_tablet_manager.cc:1428] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260929 00:58:12.695613 16863 raft_consensus.cc:348] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.696018 16863 raft_consensus.cc:374] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:12.696132 16863 raft_consensus.cc:729] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb28377a51814da190236b3e73d3b1b1, State: Initialized, Role: FOLLOWER
I20260929 00:58:12.696733 16863 consensus_queue.cc:260] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.698290 16863 ts_tablet_manager.cc:1428] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Time spent starting tablet: real 0.017s user 0.006s sys 0.006s
W20260929 00:58:12.802594 16847 tablet.cc:2369] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:12.817137 16869 raft_consensus.cc:482] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:58:12.817421 16869 raft_consensus.cc:504] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.818993 16869 leader_election.cc:288] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451), 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207)
I20260929 00:58:12.825695 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.819428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33207 (local address 127.0.97.3:42621)
0929 00:58:12.819821 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.819851 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.820030 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.820438 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.820446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.820477 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:12.820707 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.820721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.821946 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.821952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.823579 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.823591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.824935 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.824943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.825143 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.825183 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.825291 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.825435 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1}
I20260929 00:58:12.825745 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.819581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42621 (local address 127.0.97.2:33207)
0929 00:58:12.819829 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.819837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.819958 (+ 121us) server_negotiation.cc:407] Connection header received
0929 00:58:12.820188 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.820194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.820276 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.820402 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:12.820838 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.821809 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.823725 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.824777 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.824880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.824950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.825324 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.825485 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 00:58:12.826179 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.819443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37739 (local address 127.0.97.1:39451)
0929 00:58:12.819652 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.819661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.820413 (+ 752us) server_negotiation.cc:407] Connection header received
0929 00:58:12.820594 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.820607 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.820686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.820773 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:12.821396 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.822224 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.824234 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.825245 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.825368 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.825460 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.825813 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.825947 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:58:12.826592 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.819484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.97.3:37739)
0929 00:58:12.820296 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:12.820314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:12.820435 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:12.820820 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:12.820824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:12.820849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:12.821221 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.821233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.822352 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.822355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.823989 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:12.824003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.825451 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.825457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:12.825633 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.825664 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:58:12.826084 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 00:58:12.826216 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":18,"thread_start_us":479,"threads_started":1}
I20260929 00:58:12.826853 16670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c400c5144592adf08a4fe48ddbec" candidate_uuid: "fe25e771666c4a6f83b01709d78c1a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" is_pre_election: true
I20260929 00:58:12.827425 16670 raft_consensus.cc:2454] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe25e771666c4a6f83b01709d78c1a66 in term 0.
I20260929 00:58:12.827618 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c400c5144592adf08a4fe48ddbec" candidate_uuid: "fe25e771666c4a6f83b01709d78c1a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb28377a51814da190236b3e73d3b1b1" is_pre_election: true
I20260929 00:58:12.828116 16736 leader_election.cc:302] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3d01d4a26c43efbaa94fcc79f2fca0, fe25e771666c4a6f83b01709d78c1a66; no voters:
I20260929 00:58:12.828155 16538 raft_consensus.cc:2454] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe25e771666c4a6f83b01709d78c1a66 in term 0.
I20260929 00:58:12.828519 16869 raft_consensus.cc:2781] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:58:12.828696 16869 raft_consensus.cc:482] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:58:12.828810 16869 raft_consensus.cc:3037] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:12.830971 16869 raft_consensus.cc:504] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.831558 16869 leader_election.cc:288] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [CANDIDATE]: Term 1 election: Requested vote from peers bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451), 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207)
I20260929 00:58:12.831892 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c400c5144592adf08a4fe48ddbec" candidate_uuid: "fe25e771666c4a6f83b01709d78c1a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb28377a51814da190236b3e73d3b1b1"
I20260929 00:58:12.832093 16538 raft_consensus.cc:3037] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:12.832172 16670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c400c5144592adf08a4fe48ddbec" candidate_uuid: "fe25e771666c4a6f83b01709d78c1a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
I20260929 00:58:12.832355 16670 raft_consensus.cc:3037] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:12.835430 16538 raft_consensus.cc:2454] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe25e771666c4a6f83b01709d78c1a66 in term 1.
I20260929 00:58:12.835479 16670 raft_consensus.cc:2454] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe25e771666c4a6f83b01709d78c1a66 in term 1.
I20260929 00:58:12.835795 16736 leader_election.cc:302] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3d01d4a26c43efbaa94fcc79f2fca0, fe25e771666c4a6f83b01709d78c1a66; no voters:
I20260929 00:58:12.836067 16869 raft_consensus.cc:2781] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:12.836544 16869 raft_consensus.cc:686] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 1 LEADER]: Becoming Leader. State: Replica: fe25e771666c4a6f83b01709d78c1a66, State: Running, Role: LEADER
I20260929 00:58:12.837292 16869 consensus_queue.cc:237] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } }
I20260929 00:58:12.841732 16398 catalog_manager.cc:5495] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fe25e771666c4a6f83b01709d78c1a66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25e771666c4a6f83b01709d78c1a66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 } health_report { overall_health: UNKNOWN } } }
W20260929 00:58:12.843629 16716 tablet.cc:2369] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:12.892707 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.887126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.0.97.3:35335)
0929 00:58:12.887321 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.887327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.887422 (+ 95us) server_negotiation.cc:407] Connection header received
0929 00:58:12.887658 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.887662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.887734 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.887876 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:58:12.888219 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.888989 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.890503 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.890912 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.891035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.891523 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:58:12.892129 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:58:12.892279 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.892335 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.892422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 00:58:12.900761 16584 tablet.cc:2369] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:12.900959 16538 raft_consensus.cc:1261] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Refusing update from remote peer fe25e771666c4a6f83b01709d78c1a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:12.900955 16670 raft_consensus.cc:1261] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Refusing update from remote peer fe25e771666c4a6f83b01709d78c1a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:12.901993 16872 consensus_queue.cc:1035] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb28377a51814da190236b3e73d3b1b1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:12.902319 16869 consensus_queue.cc:1035] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:12.927212 16398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:12.927479 16398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:12.930624 16398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Sending DeleteTablet for 3 replicas of tablet 6cb1c400c5144592adf08a4fe48ddbec
I20260929 00:58:12.931640 16781 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52818
I20260929 00:58:12.931728 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44012
I20260929 00:58:12.932288 16650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c400c5144592adf08a4fe48ddbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54246
I20260929 00:58:12.932663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16435
I20260929 00:58:12.953750 16886 tablet_replica.cc:330] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: stopping tablet replica
I20260929 00:58:12.954173 16886 raft_consensus.cc:2229] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:12.954663 16886 raft_consensus.cc:2258] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:12.957139 16886 ts_tablet_manager.cc:1905] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:12.959210 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:12.934504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.0.97.1:39451)
0929 00:58:12.934745 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:12.934772 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:12.934795 (+ 23us) server_negotiation.cc:407] Connection header received
0929 00:58:12.934864 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:12.934871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:12.935002 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:12.935119 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:12.937641 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.939240 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.941878 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:12.942790 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:12.942932 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:12.947706 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:12.947729 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:12.947754 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:12.947819 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:12.953141 (+ 5322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.955896 (+ 2755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.955905 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.955908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.956074 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:12.956564 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:12.956569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:12.956572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:12.957072 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:12.957376 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:12.958590 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0929 00:58:12.958828 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:58:12.959892 16578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:12.961475 16885 tablet_replica.cc:330] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: stopping tablet replica
I20260929 00:58:12.961577 16887 tablet_replica.cc:330] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: stopping tablet replica
I20260929 00:58:12.962051 16887 raft_consensus.cc:2229] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:12.962214 16885 raft_consensus.cc:2229] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:58:12.962651 16887 raft_consensus.cc:2258] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:12.963263 16885 raft_consensus.cc:2258] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:12.966813 16887 ts_tablet_manager.cc:1905] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:12.970395 16885 ts_tablet_manager.cc:1905] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:12.973320 16887 ts_tablet_manager.cc:1918] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:12.973481 16887 log.cc:1196] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/6cb1c400c5144592adf08a4fe48ddbec
I20260929 00:58:12.973994 16887 ts_tablet_manager.cc:1939] T 6cb1c400c5144592adf08a4fe48ddbec P 6d3d01d4a26c43efbaa94fcc79f2fca0: Deleting consensus metadata
I20260929 00:58:12.975970 16384 catalog_manager.cc:4841] TS 6d3d01d4a26c43efbaa94fcc79f2fca0 (127.0.97.2:33207): tablet 6cb1c400c5144592adf08a4fe48ddbec (table test-table [id=60b7f1fb573e4fc6a6eab4f1cd37c98e]) successfully deleted
I20260929 00:58:12.977593 16885 ts_tablet_manager.cc:1918] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:12.977770 16885 log.cc:1196] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/6cb1c400c5144592adf08a4fe48ddbec
I20260929 00:58:12.978207 16885 ts_tablet_manager.cc:1939] T 6cb1c400c5144592adf08a4fe48ddbec P fe25e771666c4a6f83b01709d78c1a66: Deleting consensus metadata
I20260929 00:58:12.979628 16385 catalog_manager.cc:4841] TS fe25e771666c4a6f83b01709d78c1a66 (127.0.97.3:35335): tablet 6cb1c400c5144592adf08a4fe48ddbec (table test-table [id=60b7f1fb573e4fc6a6eab4f1cd37c98e]) successfully deleted
I20260929 00:58:13.083470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16435
W20260929 00:58:13.108254 16385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39451 (error 108)
I20260929 00:58:13.109025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16587
I20260929 00:58:13.110090 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.109561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44032)
0929 00:58:13.109817 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.109918 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 00:58:13.110327 16385 catalog_manager.cc:4584] TS bb28377a51814da190236b3e73d3b1b1 (127.0.97.1:39451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cb1c400c5144592adf08a4fe48ddbec: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
I20260929 00:58:13.123071 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.109749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.97.2:33207)
0929 00:58:13.109931 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:13.109937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:13.110036 (+ 99us) server_negotiation.cc:407] Connection header received
0929 00:58:13.110291 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:13.110299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:13.110403 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:13.110509 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:13.111910 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:13.112804 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:13.114106 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:13.114504 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:13.114631 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:13.117422 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:13.117435 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:13.117450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:13.117503 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:13.120228 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:13.121026 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:13.121040 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:13.121042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:13.121162 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:13.121530 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:13.121535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:13.121539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:13.121908 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:13.122205 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:13.122508 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 00:58:13.122742 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:58:13.123714 16710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:13.163894 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.163444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44042)
0929 00:58:13.163651 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.163748 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 00:58:13.164989 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.164653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44044)
0929 00:58:13.164795 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.164880 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 00:58:13.225440 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.224939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44050)
0929 00:58:13.225181 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.225274 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 00:58:13.226496 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44066)
0929 00:58:13.226302 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.226380 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 00:58:13.274853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16587
I20260929 00:58:13.292687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16718
I20260929 00:58:13.307456 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44072)
0929 00:58:13.307189 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.307266 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 00:58:13.308633 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.293592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.0.97.3:35335)
0929 00:58:13.293791 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:13.293799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:13.293830 (+ 31us) server_negotiation.cc:407] Connection header received
0929 00:58:13.293896 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:13.293902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:13.294004 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:13.294132 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:13.295411 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:13.296748 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:13.298039 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:13.298743 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:13.298884 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:13.301446 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:13.301461 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:13.301483 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:13.301525 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:13.305543 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:13.306223 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:13.306234 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:13.306238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:13.306424 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:13.306835 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:13.306840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:13.306844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:13.307312 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:13.307623 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:13.307964 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 00:58:13.308209 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:58:13.309101 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.308285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44078)
0929 00:58:13.308859 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.308951 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 00:58:13.309688 16841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:13.433315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16718
I20260929 00:58:13.453119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16365
I20260929 00:58:13.454248 16428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:13.566325 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:13.565701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39451 (local address 127.0.0.1:44092)
0929 00:58:13.565950 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:13.566079 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 00:58:13.566581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16365
I20260929 00:58:33.588984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--webserver_interface=127.0.97.62
--webserver_port=33859
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:33.920305 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:33.920678 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:33.920838 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:33.928028 16893 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 00:58:33.928123 16893 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 00:58:33.928188 16893 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 00:58:33.928260 16893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:33.928351 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:33.928460 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:33.928582 16893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:33.928670 16893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:33.938592 16893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:33.939924 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:33.941359 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:33.953297 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:33.953401 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:33.953325 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:33.954121 16893 server_base.cc:1034] running on GCE node
I20260929 00:58:33.954871 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:33.956231 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:33.957468 16893 hybrid_clock.cc:648] HybridClock initialized: now 1790643513957384 us; error 64 us; skew 500 ppm
I20260929 00:58:33.957806 16893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:33 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:33.960947 16893 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:58:33.962436 16893 webserver.cc:460] Webserver started at http://127.0.97.62:33859/ using document root <none> and password file <none>
I20260929 00:58:33.963078 16893 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:33.963160 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:33.968660 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:58:33.971361 16908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:33.972774 16893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:33.972946 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "0c90c3c984b1424181f61293beb75439"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "0bc1fcacc668aa2b3ff41ad81a2d0e59"
server_key_iv: "2230176c891faa60b3b77e93ca7b3b0d"
server_key_version: "encryptionkey@0"
I20260929 00:58:33.973857 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:34.003443 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:34.007705 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:34.008015 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:34.026897 16893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43469
I20260929 00:58:34.026944 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43469 every 8 connection(s)
I20260929 00:58:34.028435 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:34.032240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16893
I20260929 00:58:34.033253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39451
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=46345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:58:34.037139 16960 sys_catalog.cc:263] Verifying existing consensus state
I20260929 00:58:34.042434 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap starting.
I20260929 00:58:34.073530 16960 log.cc:824] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:34.094213 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 00:58:34.094821 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Bootstrap complete.
I20260929 00:58:34.104055 16960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:34.105115 16960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Initialized, Role: FOLLOWER
I20260929 00:58:34.105800 16960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:34.106040 16960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:34.106173 16960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:34.106367 16960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 2 FOLLOWER]: Advancing to term 3
I20260929 00:58:34.110278 16960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:34.110782 16960 leader_election.cc:302] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c90c3c984b1424181f61293beb75439; no voters:
I20260929 00:58:34.111415 16960 leader_election.cc:288] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 00:58:34.111620 16964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 00:58:34.112882 16964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [term 3 LEADER]: Becoming Leader. State: Replica: 0c90c3c984b1424181f61293beb75439, State: Running, Role: LEADER
I20260929 00:58:34.113464 16964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } }
I20260929 00:58:34.113668 16960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:34.116465 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c90c3c984b1424181f61293beb75439. Latest consensus state: current_term: 3 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:58:34.117275 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:34.116811 16965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0c90c3c984b1424181f61293beb75439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90c3c984b1424181f61293beb75439" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43469 } } }
I20260929 00:58:34.118130 16965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:34.128161 16972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:34.137970 16972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:34.139499 16972 catalog_manager.cc:1259] Loaded cluster ID: 4aad156ec0b2404ab2fc5bb48a29609a
I20260929 00:58:34.139653 16972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:34.143836 16972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:34.145538 16972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Loaded TSK: 0
I20260929 00:58:34.145785 16972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c90c3c984b1424181f61293beb75439: Loaded TSK: 1
F20260929 00:58:34.147120 16972 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8f0fd9b9ad google::LogMessage::Fail() at ??:0
@ 0x7f8f0fd9e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8f0fd9b4de google::LogMessage::Flush() at ??:0
@ 0x7f8f0fd9ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8f13aff7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8f13aee73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8f257ca1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8f257c9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8f258552a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f8f258550c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f8f2589f5e9 std::function<>::operator()() at ??:0
@ 0x7f8f257cdcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8f257cbc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8f25853610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8f25853439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8f129e7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8f129ea91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8f129ea799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8f129c5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8f15c9f6db start_thread at ??:0
@ 0x7f8f0ec3561f clone at ??:0
W20260929 00:58:34.455765 16962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:34.456051 16962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:34.456146 16962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:34.463589 16962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:34.463708 16962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:34.463820 16962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:58:34.473948 16962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46345
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:34.475203 16962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:34.476464 16962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:34.489558 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:34.489627 17007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:34.489648 17008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:34.490687 16962 server_base.cc:1034] running on GCE node
I20260929 00:58:34.491364 16962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:34.492350 16962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:34.493577 16962 hybrid_clock.cc:648] HybridClock initialized: now 1790643514493521 us; error 77 us; skew 500 ppm
I20260929 00:58:34.493902 16962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:34 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:34.496975 16962 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:58:34.498380 16962 webserver.cc:460] Webserver started at http://127.0.97.1:46345/ using document root <none> and password file <none>
I20260929 00:58:34.499081 16962 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:34.499236 16962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:34.504869 16962 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 00:58:34.507674 17017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:34.509117 16962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:34.509312 16962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "bb28377a51814da190236b3e73d3b1b1"
format_stamp: "Formatted at 2026-09-29 00:57:48 on dist-test-slave-pzsx"
server_key: "3c8a55c6972c8b2cd6f33c147bf18bda"
server_key_iv: "c93c4b3983ce0e008c30e6bf18027a02"
server_key_version: "encryptionkey@0"
I20260929 00:58:34.510332 16962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:34.552846 16962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:34.557018 16962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:34.557314 16962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:34.558343 16962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:34.560195 17024 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 00:58:34.565773 17024 ts_tablet_manager.cc:1852] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 00:58:34.566764 17024 ts_tablet_manager.cc:1905] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:34.574020 17024 ts_tablet_manager.cc:1918] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:34.574234 17024 log.cc:1196] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/6cb1c400c5144592adf08a4fe48ddbec
I20260929 00:58:34.574797 17024 ts_tablet_manager.cc:1939] T 6cb1c400c5144592adf08a4fe48ddbec P bb28377a51814da190236b3e73d3b1b1: Deleting consensus metadata
I20260929 00:58:34.575203 16962 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 00:58:34.575340 16962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260929 00:58:34.575508 16962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:34.575608 16962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:34.609251 16962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39451
I20260929 00:58:34.609313 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39451 every 8 connection(s)
I20260929 00:58:34.610611 16962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:58:34.615615 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:34.613697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:59823)
0929 00:58:34.614500 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:34.614736 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":309,"threads_started":1}
I20260929 00:58:34.615832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 16962
W20260929 00:58:34.616255 17131 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43469 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43469: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
I20260929 00:58:34.616597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33207
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=32861
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:58:34.617177 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:34.616845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:45061)
0929 00:58:34.617002 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:34.617080 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:58:34.618148 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:34.617697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:53057)
0929 00:58:34.617880 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:34.617984 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 00:58:34.618388 17131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:58:34.943158 17134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:34.943490 17134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:34.943683 17134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:34.950826 17134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:34.951007 17134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:34.951211 17134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:58:34.960916 17134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32861
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:34.962234 17134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:34.963617 17134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:34.976580 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:34.976625 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:34.976650 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:34.977331 17134 server_base.cc:1034] running on GCE node
I20260929 00:58:34.977859 17134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:34.978705 17134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:34.979928 17134 hybrid_clock.cc:648] HybridClock initialized: now 1790643514979887 us; error 51 us; skew 500 ppm
I20260929 00:58:34.980252 17134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:34 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:34.983227 17134 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:58:34.984709 17134 webserver.cc:460] Webserver started at http://127.0.97.2:32861/ using document root <none> and password file <none>
I20260929 00:58:34.985436 17134 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:34.985579 17134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:34.990662 17134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 00:58:34.993240 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:34.994470 17134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:34.994648 17134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "6d3d01d4a26c43efbaa94fcc79f2fca0"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "f8d31e58a4941a3b833d88aec1b9782a"
server_key_iv: "a13aeb9ea38558ed805dfd19ea14dc3d"
server_key_version: "encryptionkey@0"
I20260929 00:58:34.995630 17134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:35.010830 17134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:35.015252 17134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:35.015539 17134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:35.016585 17134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:35.017907 17134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:35.018047 17134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:35.018174 17134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:35.018262 17134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:35.051431 17134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33207
I20260929 00:58:35.051515 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33207 every 8 connection(s)
I20260929 00:58:35.052953 17134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:58:35.057860 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.056057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:35567)
0929 00:58:35.056828 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.057028 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":283,"threads_started":1}
W20260929 00:58:35.058562 17262 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43469 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43469: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
I20260929 00:58:35.059253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17134
I20260929 00:58:35.059473 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.059127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:59119)
0929 00:58:35.059272 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.059336 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 00:58:35.059998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35335
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43469
--webserver_port=41391
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:58:35.060443 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.060027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:37001)
0929 00:58:35.060210 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.060297 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:58:35.060738 17262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 00:58:35.386797 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:35.387112 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:35.387341 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:35.394636 17265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:35.394742 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:35.394891 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:58:35.404698 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41391
--tserver_master_addrs=127.0.97.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:35.405961 17265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:35.407272 17265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:35.420686 17273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:35.420796 17271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:35.420709 17270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:35.426087 17265 server_base.cc:1034] running on GCE node
I20260929 00:58:35.426730 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:35.427671 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:35.429000 17265 hybrid_clock.cc:648] HybridClock initialized: now 1790643515428927 us; error 62 us; skew 500 ppm
I20260929 00:58:35.429330 17265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:35 dist-test-slave-pzsx krb5kdc[15813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:35.439885 17265 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:58:35.441921 17265 webserver.cc:460] Webserver started at http://127.0.97.3:41391/ using document root <none> and password file <none>
I20260929 00:58:35.442993 17265 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:35.443135 17265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:35.448747 17265 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 00:58:35.451444 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:35.452979 17265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:35.453157 17265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "fe25e771666c4a6f83b01709d78c1a66"
format_stamp: "Formatted at 2026-09-29 00:57:49 on dist-test-slave-pzsx"
server_key: "9e3721050d99c783237783bbc72e935a"
server_key_iv: "8666db5eb87e25ca59506092254c8ffb"
server_key_version: "encryptionkey@0"
I20260929 00:58:35.454125 17265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:35.467929 17265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:35.471767 17265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:35.472056 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:35.473104 17265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:35.474378 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:35.474498 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:35.474628 17265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:35.474727 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:35.508169 17265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35335
I20260929 00:58:35.508261 17392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35335 every 8 connection(s)
I20260929 00:58:35.509675 17265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:58:35.512898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17265
I20260929 00:58:35.514868 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.512591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:41657)
0929 00:58:35.513521 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.513764 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":429,"threads_started":1}
W20260929 00:58:35.515530 17393 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43469 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43469: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
I20260929 00:58:35.516326 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.516010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:40865)
0929 00:58:35.516155 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.516226 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 00:58:35.517146 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.516837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:34549)
0929 00:58:35.516968 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.517045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:58:35.517381 17393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 00:58:35.619733 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:35.618922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:60767)
0929 00:58:35.619397 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:35.619490 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
I20260929 00:58:36.062000 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:36.061271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:42671)
0929 00:58:36.061692 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:36.061774 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":221,"threads_started":1}
I20260929 00:58:36.518647 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:36.517873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:57997)
0929 00:58:36.518327 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:36.518413 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":223,"threads_started":1}
I20260929 00:58:36.621424 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:36.620730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:49989)
0929 00:58:36.621167 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:36.621269 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
I20260929 00:58:37.063612 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:37.062833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:46535)
0929 00:58:37.063254 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:37.063396 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260929 00:58:37.520249 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:37.519536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:39157)
0929 00:58:37.519981 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:37.520069 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":229,"threads_started":1}
I20260929 00:58:37.623111 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:37.622428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:34657)
0929 00:58:37.622844 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:37.622928 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260929 00:58:38.065233 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:38.064568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:51617)
0929 00:58:38.064988 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:38.065067 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":218,"threads_started":1}
I20260929 00:58:38.521834 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:38.521159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:56253)
0929 00:58:38.521585 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:38.521664 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260929 00:58:38.624836 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:38.624105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:44483)
0929 00:58:38.624593 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:38.624679 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260929 00:58:39.066927 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:39.066190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:49753)
0929 00:58:39.066650 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:39.066734 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":219,"threads_started":1}
I20260929 00:58:39.523483 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:39.522783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:49049)
0929 00:58:39.523237 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:39.523317 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260929 00:58:39.626583 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:39.625727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:48617)
0929 00:58:39.626327 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:39.626401 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":367,"threads_started":1}
I20260929 00:58:40.068853 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:40.067932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:52693)
0929 00:58:40.068580 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:40.068681 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":317,"threads_started":1}
I20260929 00:58:40.525174 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:40.524368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:50557)
0929 00:58:40.524913 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:40.524989 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":329,"threads_started":1}
I20260929 00:58:40.628100 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:40.627458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:41355)
0929 00:58:40.627863 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:40.627939 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260929 00:58:41.070513 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:41.069821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:37979)
0929 00:58:41.070264 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:41.070351 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260929 00:58:41.526774 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:41.526107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:60131)
0929 00:58:41.526525 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:41.526610 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260929 00:58:41.629817 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:41.629015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:37623)
0929 00:58:41.629520 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:41.629600 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":231,"threads_started":1}
I20260929 00:58:42.072263 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:42.071445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:45485)
0929 00:58:42.071967 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:42.072047 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":241,"threads_started":1}
I20260929 00:58:42.528446 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:42.527668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:46329)
0929 00:58:42.528156 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:42.528237 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":227,"threads_started":1}
I20260929 00:58:42.631330 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:42.630693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:49343)
0929 00:58:42.631092 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:42.631169 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":208,"threads_started":1}
I20260929 00:58:43.073921 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:43.073261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:44639)
0929 00:58:43.073687 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:43.073767 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260929 00:58:43.530277 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:43.529387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:48029)
0929 00:58:43.529984 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:43.530074 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":356,"threads_started":1}
I20260929 00:58:43.633029 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:43.632264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:51571)
0929 00:58:43.632751 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:43.632850 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260929 00:58:44.075453 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:44.074768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:39887)
0929 00:58:44.075214 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:44.075290 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260929 00:58:44.531948 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:44.531190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:58011)
0929 00:58:44.531681 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:44.531761 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
I20260929 00:58:44.634677 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:44.633952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:42625)
0929 00:58:44.634432 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:44.634513 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260929 00:58:45.077308 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:45.076649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:60593)
0929 00:58:45.077061 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:45.077149 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":222,"threads_started":1}
I20260929 00:58:45.533612 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:45.532964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:33847)
0929 00:58:45.533376 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:45.533453 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":235,"threads_started":1}
I20260929 00:58:45.636463 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:45.635699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:35199)
0929 00:58:45.636190 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:45.636279 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260929 00:58:46.078872 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:46.078172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:52019)
0929 00:58:46.078572 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:46.078697 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":198,"threads_started":1}
I20260929 00:58:46.535355 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:46.534627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:37867)
0929 00:58:46.535086 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:46.535177 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260929 00:58:46.638262 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:46.637534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:40605)
0929 00:58:46.637952 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:46.638031 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":216,"threads_started":1}
I20260929 00:58:47.080399 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:47.079721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:49115)
0929 00:58:47.080143 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:47.080230 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260929 00:58:47.537132 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:47.536387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:50093)
0929 00:58:47.536897 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:47.536982 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
I20260929 00:58:47.640201 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:47.639432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:58377)
0929 00:58:47.639911 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:47.639993 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260929 00:58:48.082046 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:48.081292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:39773)
0929 00:58:48.081767 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:48.081856 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260929 00:58:48.538868 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:48.538123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:37087)
0929 00:58:48.538592 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:48.538681 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260929 00:58:48.641944 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:48.641223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:39911)
0929 00:58:48.641675 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:48.641754 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260929 00:58:49.083642 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:49.082907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:54997)
0929 00:58:49.083313 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:49.083444 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260929 00:58:49.540695 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:49.539806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:53131)
0929 00:58:49.540347 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:49.540464 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":307,"threads_started":1}
I20260929 00:58:49.643744 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:49.643024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.1:34995)
0929 00:58:49.643479 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:49.643556 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
I20260929 00:58:50.085579 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:50.084805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.2:47075)
0929 00:58:50.085312 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:50.085397 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":292,"threads_started":1}
I20260929 00:58:50.515580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 16962
I20260929 00:58:50.534637 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:50.517513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.1:39451)
0929 00:58:50.518080 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:50.518103 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:50.518146 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:58:50.519787 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:50.519798 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:50.520243 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:50.520613 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:50.522042 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.523302 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.524652 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.525320 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.525528 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:50.528240 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:50.528249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:50.528265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:50.528315 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:50.531195 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.531901 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.531909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.531915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.532088 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.532470 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.532475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.532478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.533108 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:50.533453 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:50.533884 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 00:58:50.534327 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":289,"threads_started":1}
I20260929 00:58:50.536166 17126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:50.542690 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:50.541843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43469 (local address 127.0.97.3:58447)
0929 00:58:50.542334 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:50.542446 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260929 00:58:50.625239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 16962
I20260929 00:58:50.643283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17134
I20260929 00:58:50.661057 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:50.644298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.0.97.2:33207)
0929 00:58:50.644870 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:50.644884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:50.644920 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:58:50.646708 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:50.646718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:50.647165 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:50.647457 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:50.648882 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.650151 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.651360 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.652001 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.652146 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:50.654660 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:50.654668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:50.654683 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:50.654719 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:50.658020 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.658610 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.658617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.658623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.658838 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.659151 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.659154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.659156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.659660 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:50.659891 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:50.660194 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 00:58:50.660682 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
I20260929 00:58:50.662408 17257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:50.747634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17134
I20260929 00:58:50.765077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17265
I20260929 00:58:50.782395 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:50.766171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.97.3:35335)
0929 00:58:50.766773 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:50.766792 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:50.766835 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:58:50.768447 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:50.768456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:50.769002 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:50.769310 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:50.770567 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.771741 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.773094 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:50.773731 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:50.773865 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:50.776350 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:50.776357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:50.776372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:50.776406 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:50.779461 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.780066 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.780072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.780078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.780247 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:50.780601 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:50.780606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:50.780608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:50.781127 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:50.781350 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:50.781642 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 00:58:50.782079 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":288,"threads_started":1}
I20260929 00:58:50.783882 17388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:50.868435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17265
2026-09-29T00:58:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64392 ms)
[----------] 30 tests from SecurityITest (292980 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:58:50 dist-test-slave-pzsx krb5kdc[17450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:58:50 dist-test-slave-pzsx krb5kdc[17450](info): set up 2 sockets
Sep 29 00:58:50 dist-test-slave-pzsx krb5kdc[17450](info): commencing operation
krb5kdc: starting...
W20260929 00:58:51.964766 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:58:52 dist-test-slave-pzsx krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:58:52.201756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41777
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:52.532959 17463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:52.533234 17463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:52.533321 17463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:52.540261 17463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:52.540359 17463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:52.540405 17463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:52.540462 17463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:52.540573 17463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:52.550175 17463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:52.551352 17463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:52.552711 17463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:52.564227 17468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:52.564270 17469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:52.564129 17471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:52.565274 17463 server_base.cc:1034] running on GCE node
I20260929 00:58:52.565997 17463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:52.567317 17463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:52.568595 17463 hybrid_clock.cc:648] HybridClock initialized: now 1790643532568541 us; error 50 us; skew 500 ppm
I20260929 00:58:52.568935 17463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:52 dist-test-slave-pzsx krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:52.571835 17463 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:58:52.573374 17463 webserver.cc:460] Webserver started at http://127.0.97.62:40953/ using document root <none> and password file <none>
I20260929 00:58:52.574038 17463 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:52.574177 17463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:52.574474 17463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:52.577323 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "3b2bebf6fc3048ca9a306efe74166767"
format_stamp: "Formatted at 2026-09-29 00:58:52 on dist-test-slave-pzsx"
server_key: "7b8eba2f49e448e7ded6931f872e458f"
server_key_iv: "9eeb0315e46db9c7dd02e6c37eed029f"
server_key_version: "encryptionkey@0"
I20260929 00:58:52.578051 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b2bebf6fc3048ca9a306efe74166767"
format_stamp: "Formatted at 2026-09-29 00:58:52 on dist-test-slave-pzsx"
server_key: "7b8eba2f49e448e7ded6931f872e458f"
server_key_iv: "9eeb0315e46db9c7dd02e6c37eed029f"
server_key_version: "encryptionkey@0"
I20260929 00:58:52.582919 17463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 00:58:52.586177 17478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:52.587378 17463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 00:58:52.587558 17463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "3b2bebf6fc3048ca9a306efe74166767"
format_stamp: "Formatted at 2026-09-29 00:58:52 on dist-test-slave-pzsx"
server_key: "7b8eba2f49e448e7ded6931f872e458f"
server_key_iv: "9eeb0315e46db9c7dd02e6c37eed029f"
server_key_version: "encryptionkey@0"
I20260929 00:58:52.587762 17463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:52.611604 17463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:52.615610 17463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:52.615914 17463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:52.634552 17463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41777
I20260929 00:58:52.634604 17529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41777 every 8 connection(s)
I20260929 00:58:52.635911 17463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:52.639673 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:52.643756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17463
I20260929 00:58:52.644001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:58:52.644709 388 external_mini_cluster.cc:1428] Setting key 51a4900563ce62cdf4fcb935ad046fa5
I20260929 00:58:52.650646 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: Bootstrap starting.
I20260929 00:58:52.654361 17530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:52.655541 17530 log.cc:824] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:52.658937 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: No bootstrap required, opened a new log
Sep 29 00:58:52 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:52.667148 17530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260929 00:58:52.667538 17530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:52.667662 17530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2bebf6fc3048ca9a306efe74166767, State: Initialized, Role: FOLLOWER
I20260929 00:58:52.668056 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:52.646738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.97.62:41777)
0929 00:58:52.647647 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:52.647660 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:52.647700 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:58:52.649531 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:52.649554 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:52.649924 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:52.650247 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:52.652017 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:52.653424 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:52.655280 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:52.656007 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:52.656241 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:52.660801 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:52.660809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:52.660822 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:52.660861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:52.663917 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:52.664629 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:52.664639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:52.664651 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:52.664839 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:52.665237 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:52.665241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:52.665251 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:52.665772 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:52.666085 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:52.666490 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 00:58:52.666902 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":484,"threads_started":1}
I20260929 00:58:52.668282 17530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260929 00:58:52.668572 17530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:52.668749 17530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:52.668937 17530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:52.671965 17530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260929 00:58:52.672477 17530 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2bebf6fc3048ca9a306efe74166767; no voters:
I20260929 00:58:52.673130 17530 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:58:52.673255 17535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:52.673990 17535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2bebf6fc3048ca9a306efe74166767, State: Running, Role: LEADER
I20260929 00:58:52.674559 17535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } }
I20260929 00:58:52.675225 17530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:52.677605 17537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2bebf6fc3048ca9a306efe74166767. Latest consensus state: current_term: 1 leader_uuid: "3b2bebf6fc3048ca9a306efe74166767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } } }
I20260929 00:58:52.678052 17537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:52.678851 17536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2bebf6fc3048ca9a306efe74166767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2bebf6fc3048ca9a306efe74166767" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41777 } } }
I20260929 00:58:52.681021 17536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767 [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:52.682265 17543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:52.686527 17543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:52.695586 17543 catalog_manager.cc:1347] Generated new cluster ID: 1f4a349429ab4d02a6c34c205ff214c8
I20260929 00:58:52.695698 17543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:52.703524 17543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:58:52.705368 17543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:52.715272 17543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: Generated new TSK 0
I20260929 00:58:52.716269 17543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:58:52.769755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:53.099032 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:53.099310 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:53.099471 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:53.107023 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:53.107158 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:53.107342 17556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:58:53.117072 17556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:53.118405 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:53.119803 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:53.132838 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:53.132838 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:53.132791 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:53.133751 17556 server_base.cc:1034] running on GCE node
I20260929 00:58:53.134302 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:53.135166 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:53.136391 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790643533136350 us; error 35 us; skew 500 ppm
I20260929 00:58:53.136742 17556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:53 dist-test-slave-pzsx krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:53.139830 17556 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:58:53.141388 17556 webserver.cc:460] Webserver started at http://127.0.97.1:37681/ using document root <none> and password file <none>
I20260929 00:58:53.142162 17556 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:53.142309 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:53.142627 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:53.145676 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "0dd11e32ce41439bb3c725959377c059"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "47cf331dfc41d69c19143bbfe1f231c3"
server_key_iv: "140435f5ef8f39485dc2556159f8980a"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.146494 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0dd11e32ce41439bb3c725959377c059"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "47cf331dfc41d69c19143bbfe1f231c3"
server_key_iv: "140435f5ef8f39485dc2556159f8980a"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.151772 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 00:58:53.155422 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.156910 17556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:58:53.157107 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "0dd11e32ce41439bb3c725959377c059"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "47cf331dfc41d69c19143bbfe1f231c3"
server_key_iv: "140435f5ef8f39485dc2556159f8980a"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.157320 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:53.175956 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:53.179960 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:53.180212 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:53.181205 17556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:53.182550 17556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:53.182621 17556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.182832 17556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:53.182879 17556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.216746 17556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38255
I20260929 00:58:53.216770 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38255 every 8 connection(s)
I20260929 00:58:53.218043 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:58:53.223419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17556
I20260929 00:58:53.223618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:58:53.224231 388 external_mini_cluster.cc:1428] Setting key 6de51937d66bfcb6333e1195cbd81be9
Sep 29 00:58:53 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:53.239619 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:53.220557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34395 (local address 127.0.97.62:41777)
0929 00:58:53.220998 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:53.221002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:53.222020 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 00:58:53.224023 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:53.224030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:53.224122 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:53.224243 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:53.226046 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.226749 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.228286 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.228764 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.228902 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:53.232912 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:53.232918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:53.232921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:53.232954 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:53.236342 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:53.237116 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:53.237125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:53.237132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:53.237279 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:53.237676 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:53.237684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:53.237691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:53.238076 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:53.238226 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:53.238885 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 00:58:53.239137 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":217,"threads_started":1}
I20260929 00:58:53.240207 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:53.221146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41777 (local address 127.0.97.1:34395)
0929 00:58:53.221849 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:53.221898 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:53.223743 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:53.224483 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:53.224540 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:53.225145 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:53.225762 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:53.225801 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.226894 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.226900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:53.228145 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:53.228157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.228994 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.229002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:53.229244 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:53.229933 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:53.229958 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:53.232697 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:53.236479 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:53.236485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:53.236538 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:53.236981 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:53.237383 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:53.237386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:53.237388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:53.237558 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:53.238192 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:53.238197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:53.238495 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 00:58:53.238723 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 00:58:53.239091 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":279,"threads_started":1}
I20260929 00:58:53.241855 17684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41777
I20260929 00:58:53.242187 17684 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:53.243067 17684 heartbeater.cc:511] Master 127.0.97.62:41777 requested a full tablet report, sending...
I20260929 00:58:53.245361 17495 ts_manager.cc:194] Registered new tserver with Master: 0dd11e32ce41439bb3c725959377c059 (127.0.97.1:38255)
I20260929 00:58:53.246874 17495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34395
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:58:53.279341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:53.602548 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:53.602818 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:53.603003 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:53.610472 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:53.610596 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:53.610792 17690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:58:53.620324 17690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:53.621593 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:53.622916 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:53.635917 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:53.635998 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:53.636108 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:53.637838 17690 server_base.cc:1034] running on GCE node
I20260929 00:58:53.638495 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:53.639578 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:53.640807 17690 hybrid_clock.cc:648] HybridClock initialized: now 1790643533640738 us; error 57 us; skew 500 ppm
I20260929 00:58:53.641153 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:53 dist-test-slave-pzsx krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:53.644825 17690 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:58:53.651690 17690 webserver.cc:460] Webserver started at http://127.0.97.2:34933/ using document root <none> and password file <none>
I20260929 00:58:53.652417 17690 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:53.652566 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:53.652846 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:53.655640 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "bc7e146d4ffa4f30839766b820a4cfc3"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "6e15bbeb594b2c81b36ef6915861d2b0"
server_key_iv: "e9d752802ff7c204ee613c4f6ab4f08c"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.656373 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bc7e146d4ffa4f30839766b820a4cfc3"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "6e15bbeb594b2c81b36ef6915861d2b0"
server_key_iv: "e9d752802ff7c204ee613c4f6ab4f08c"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.661576 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 00:58:53.665059 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.666386 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:58:53.666559 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "bc7e146d4ffa4f30839766b820a4cfc3"
format_stamp: "Formatted at 2026-09-29 00:58:53 on dist-test-slave-pzsx"
server_key: "6e15bbeb594b2c81b36ef6915861d2b0"
server_key_iv: "e9d752802ff7c204ee613c4f6ab4f08c"
server_key_version: "encryptionkey@0"
I20260929 00:58:53.666760 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:53.681913 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:53.686017 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:53.686340 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:53.687291 17690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:53.688594 17690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:53.688704 17690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.688848 17690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:53.688925 17690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:53.722198 17690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40085
I20260929 00:58:53.722242 17817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40085 every 8 connection(s)
I20260929 00:58:53.723503 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:58:53.732112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17690
I20260929 00:58:53.732308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:58:53.732810 388 external_mini_cluster.cc:1428] Setting key 443f91c1736106ab9944dcbb724bf89a
Sep 29 00:58:53 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:53.743970 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:53.726122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44143 (local address 127.0.97.62:41777)
0929 00:58:53.726309 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:53.726313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:53.727480 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 00:58:53.729411 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:53.729417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:53.729526 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:53.729637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:53.731106 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.731807 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.733572 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.734044 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.734166 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:53.738282 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:53.738288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:53.738291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:53.738325 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:53.740898 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:53.741649 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:53.741656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:53.741665 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:53.741788 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:53.742289 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:53.742292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:53.742294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:53.742559 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:53.742710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:53.743406 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 00:58:53.743633 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:58:53.745043 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:53.726628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41777 (local address 127.0.97.2:44143)
0929 00:58:53.727315 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:53.727362 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:53.729147 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:53.729870 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:53.729878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:53.730372 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:53.730925 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:53.730941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.731958 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.731968 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:53.733416 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:53.733427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:53.734301 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:53.734309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:53.734593 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:53.735295 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:53.735380 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:53.738044 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:53.741057 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:53.741066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:53.741083 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:53.741502 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:53.741900 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:53.741907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:53.741910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:53.742158 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:53.742696 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:53.742703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:53.743045 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 00:58:53.743302 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 00:58:53.743849 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
I20260929 00:58:53.746497 17818 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41777
I20260929 00:58:53.746795 17818 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:53.747529 17818 heartbeater.cc:511] Master 127.0.97.62:41777 requested a full tablet report, sending...
I20260929 00:58:53.749107 17495 ts_manager.cc:194] Registered new tserver with Master: bc7e146d4ffa4f30839766b820a4cfc3 (127.0.97.2:40085)
I20260929 00:58:53.750355 17495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44143
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:58:53.788815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:54.123605 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:54.123878 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:54.123971 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:54.131385 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:54.131496 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:54.131609 17823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:58:54.141165 17823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:54.142360 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:54.143622 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:54.156749 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:54.156776 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:54.156809 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:54.157660 17823 server_base.cc:1034] running on GCE node
I20260929 00:58:54.158245 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:54.159189 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:54.160409 17823 hybrid_clock.cc:648] HybridClock initialized: now 1790643534160361 us; error 44 us; skew 500 ppm
I20260929 00:58:54.160737 17823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:54 dist-test-slave-pzsx krb5kdc[17450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:54.163865 17823 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:58:54.165401 17823 webserver.cc:460] Webserver started at http://127.0.97.3:42563/ using document root <none> and password file <none>
I20260929 00:58:54.166036 17823 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:54.166133 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:54.166369 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:54.169237 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5"
format_stamp: "Formatted at 2026-09-29 00:58:54 on dist-test-slave-pzsx"
server_key: "68e2a81bbe3161c8067421201a1cc6d6"
server_key_iv: "61cdcde580f0daa628b7972320009b08"
server_key_version: "encryptionkey@0"
I20260929 00:58:54.170063 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5"
format_stamp: "Formatted at 2026-09-29 00:58:54 on dist-test-slave-pzsx"
server_key: "68e2a81bbe3161c8067421201a1cc6d6"
server_key_iv: "61cdcde580f0daa628b7972320009b08"
server_key_version: "encryptionkey@0"
I20260929 00:58:54.175164 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:58:54.178752 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:54.180058 17823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:58:54.180230 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5"
format_stamp: "Formatted at 2026-09-29 00:58:54 on dist-test-slave-pzsx"
server_key: "68e2a81bbe3161c8067421201a1cc6d6"
server_key_iv: "61cdcde580f0daa628b7972320009b08"
server_key_version: "encryptionkey@0"
I20260929 00:58:54.180431 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:54.207203 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:54.211444 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:54.211773 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:54.212841 17823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:54.214210 17823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:54.214284 17823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:54.214434 17823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:54.214484 17823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:54.248642 17823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45469
I20260929 00:58:54.248687 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45469 every 8 connection(s)
I20260929 00:58:54.249924 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:58:54.250412 17684 heartbeater.cc:503] Master 127.0.97.62:41777 was elected leader, sending a full tablet report...
I20260929 00:58:54.252583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 17823
I20260929 00:58:54.252771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:58:54.253298 388 external_mini_cluster.cc:1428] Setting key 42c88231941b4be22c5e0b0a3036ecfc
Sep 29 00:58:54 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:54.270249 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.252646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42877 (local address 127.0.97.62:41777)
0929 00:58:54.253187 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.253193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.254360 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 00:58:54.256281 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.256286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.256349 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.256439 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:54.257926 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.258684 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.260155 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.260590 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.260709 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.264036 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:54.264041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:54.264045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:54.264076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:54.267167 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.268044 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.268047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.268050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.268191 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.268697 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.268701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.268703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.268951 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:54.269078 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.269744 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.269900 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
I20260929 00:58:54.271229 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.253324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41777 (local address 127.0.97.3:42877)
0929 00:58:54.254204 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.254255 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.256034 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.256720 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.256728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.257227 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:54.257786 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.257802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.258810 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.258814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.260031 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.260046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.260807 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.260812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.261004 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.261693 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:58:54.261714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:54.263838 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:54.267306 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:54.267315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:54.267332 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:54.267891 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:54.268311 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:58:54.268327 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:58:54.268331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:58:54.268598 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:58:54.269074 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:54.269081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:54.269379 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.269643 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.270033 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":369,"threads_started":1}
I20260929 00:58:54.272820 17951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41777
I20260929 00:58:54.273147 17951 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:54.273867 17951 heartbeater.cc:511] Master 127.0.97.62:41777 requested a full tablet report, sending...
I20260929 00:58:54.275246 17495 ts_manager.cc:194] Registered new tserver with Master: d0bd73cb1cd2432c9a184b9a3328f2d5 (127.0.97.3:45469)
I20260929 00:58:54.276121 17495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42877
I20260929 00:58:54.280649 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:58:54.297307 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.285291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.97.62:41777)
0929 00:58:54.285515 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.285524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.285936 (+ 412us) server_negotiation.cc:407] Connection header received
0929 00:58:54.286051 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.286055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.286114 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.286188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:54.287470 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.288145 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.289375 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.289787 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.289878 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.292314 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:54.292319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:54.292322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:54.292353 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:54.295030 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.295633 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.295636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.295638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.295761 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.296066 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.296069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.296071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.296358 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:54.296597 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.296846 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.297000 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:58:54.301002 17495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:58:54.307665 17495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:58:54.346107 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.320548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.97.1:38255)
0929 00:58:54.337076 (+ 16528us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.337088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.337931 (+ 843us) server_negotiation.cc:407] Connection header received
0929 00:58:54.338391 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.338397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.338585 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.338748 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:54.339933 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341253 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.343387 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.344650 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.344771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.345006 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.345595 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.345766 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16328,"thread_start_us":16153,"threads_started":1}
I20260929 00:58:54.346163 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.319851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38255 (local address 127.0.0.1:33186)
0929 00:58:54.337807 (+ 17956us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.337831 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.338202 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.339033 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.339038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.339081 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:54.339711 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.339724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341435 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.341440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.343235 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.343247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.345128 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.345137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.345368 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.345404 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.345550 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.345735 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17852,"mutex_wait_us":52,"spinlock_wait_cycles":8832,"thread_start_us":17457,"threads_started":1}
I20260929 00:58:54.347288 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.319509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.97.2:40085)
0929 00:58:54.330029 (+ 10520us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.330042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.337317 (+ 7275us) server_negotiation.cc:407] Connection header received
0929 00:58:54.338353 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.338361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.338586 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.338748 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:54.339900 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341443 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.344364 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.345834 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.345956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.346141 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.346732 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.346930 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10307,"thread_start_us":10148,"threads_started":1}
I20260929 00:58:54.347491 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.319330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45469 (local address 127.0.0.1:56222)
0929 00:58:54.337568 (+ 18238us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.337591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.337889 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.338880 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.338886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.338908 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:54.339269 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.339283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341297 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.341302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.344142 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.344170 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.346692 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.346699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.346859 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.346890 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.346979 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.347098 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18112,"mutex_wait_us":175,"thread_start_us":18198,"threads_started":1}
I20260929 00:58:54.347909 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.320170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.97.3:45469)
0929 00:58:54.333001 (+ 12831us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.333012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.337711 (+ 4699us) server_negotiation.cc:407] Connection header received
0929 00:58:54.338166 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.338175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.338456 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.338599 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:54.339482 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341135 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.344364 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.345834 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.345956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.346173 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.347056 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.347216 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12625,"thread_start_us":12852,"threads_started":1}
I20260929 00:58:54.348071 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.319086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40085 (local address 127.0.0.1:53372)
0929 00:58:54.337052 (+ 17966us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.337131 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.338128 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.339033 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.339038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.339080 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:54.339722 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.339730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.341604 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.341613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.344143 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.344173 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.346314 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.346322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.346471 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.346535 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.347547 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.347691 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17601,"mutex_wait_us":162,"spinlock_wait_cycles":8320,"thread_start_us":18857,"threads_started":1}
I20260929 00:58:54.350404 17619 tablet_service.cc:1467] Processing CreateTablet for tablet ba370a11c7914fd683e3b8b460088ba6 (DEFAULT_TABLE table=test-table [id=917ed36133464ee3a294b810cf1b87df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:54.351804 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba370a11c7914fd683e3b8b460088ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:54.351383 17753 tablet_service.cc:1467] Processing CreateTablet for tablet ba370a11c7914fd683e3b8b460088ba6 (DEFAULT_TABLE table=test-table [id=917ed36133464ee3a294b810cf1b87df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:54.351379 17886 tablet_service.cc:1467] Processing CreateTablet for tablet ba370a11c7914fd683e3b8b460088ba6 (DEFAULT_TABLE table=test-table [id=917ed36133464ee3a294b810cf1b87df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:54.352733 17753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba370a11c7914fd683e3b8b460088ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:54.352731 17886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba370a11c7914fd683e3b8b460088ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:54.363278 17970 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Bootstrap starting.
I20260929 00:58:54.366598 17970 tablet_bootstrap.cc:654] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:54.367791 17971 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Bootstrap starting.
I20260929 00:58:54.368067 17970 log.cc:824] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:54.369884 17972 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Bootstrap starting.
I20260929 00:58:54.371526 17970 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: No bootstrap required, opened a new log
I20260929 00:58:54.371583 17971 tablet_bootstrap.cc:654] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:54.371814 17970 ts_tablet_manager.cc:1397] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 00:58:54.372843 17971 log.cc:824] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:54.373728 17972 tablet_bootstrap.cc:654] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:54.374940 17972 log.cc:824] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:54.376109 17971 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: No bootstrap required, opened a new log
I20260929 00:58:54.376427 17971 ts_tablet_manager.cc:1397] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 00:58:54.379762 17970 raft_consensus.cc:348] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.380118 17970 raft_consensus.cc:374] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:54.380237 17970 raft_consensus.cc:729] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd11e32ce41439bb3c725959377c059, State: Initialized, Role: FOLLOWER
I20260929 00:58:54.380916 17970 consensus_queue.cc:260] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.382634 17970 ts_tablet_manager.cc:1428] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 00:58:54.384397 17972 tablet_bootstrap.cc:492] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: No bootstrap required, opened a new log
I20260929 00:58:54.384697 17972 ts_tablet_manager.cc:1397] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260929 00:58:54.387590 17971 raft_consensus.cc:348] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.388025 17971 raft_consensus.cc:374] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:54.388118 17971 raft_consensus.cc:729] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7e146d4ffa4f30839766b820a4cfc3, State: Initialized, Role: FOLLOWER
I20260929 00:58:54.388741 17971 consensus_queue.cc:260] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.393133 17818 heartbeater.cc:503] Master 127.0.97.62:41777 was elected leader, sending a full tablet report...
I20260929 00:58:54.393800 17971 ts_tablet_manager.cc:1428] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260929 00:58:54.395597 17972 raft_consensus.cc:348] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.395958 17972 raft_consensus.cc:374] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:54.396052 17972 raft_consensus.cc:729] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bd73cb1cd2432c9a184b9a3328f2d5, State: Initialized, Role: FOLLOWER
I20260929 00:58:54.396621 17972 consensus_queue.cc:260] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.397645 17951 heartbeater.cc:503] Master 127.0.97.62:41777 was elected leader, sending a full tablet report...
I20260929 00:58:54.398171 17972 ts_tablet_manager.cc:1428] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 00:58:54.442853 17977 raft_consensus.cc:482] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:58:54.443122 17977 raft_consensus.cc:504] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.444947 17977 leader_election.cc:288] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0bd73cb1cd2432c9a184b9a3328f2d5 (127.0.97.3:45469), 0dd11e32ce41439bb3c725959377c059 (127.0.97.1:38255)
I20260929 00:58:54.451224 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.445240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52345 (local address 127.0.97.3:45469)
0929 00:58:54.445460 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.445468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.446090 (+ 622us) server_negotiation.cc:407] Connection header received
0929 00:58:54.446334 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.446338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.446397 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.446494 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:54.446985 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.447718 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.449573 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.450329 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.450483 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.450591 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.450848 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.450971 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:58:54.451408 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.445192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45469 (local address 127.0.97.2:52345)
0929 00:58:54.445965 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.445990 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.446179 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.446514 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.446519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.446545 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:54.446849 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.446857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.447848 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.447854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.449404 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.449415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.450544 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.450550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.450694 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.450716 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.450827 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.450955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":388,"threads_started":1}
I20260929 00:58:54.451545 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.445747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41843 (local address 127.0.97.1:38255)
0929 00:58:54.445997 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.446005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.446471 (+ 466us) server_negotiation.cc:407] Connection header received
0929 00:58:54.446673 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.446676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.446733 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.446830 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:54.447395 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.448088 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.449841 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.450625 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.450781 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.450886 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.451175 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.451297 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 00:58:54.451905 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.445344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38255 (local address 127.0.97.2:41843)
0929 00:58:54.446358 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:54.446372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:54.446520 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:54.446852 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:54.446856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:54.446881 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:54.447241 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.447249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.448225 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.448231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.449679 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:54.449690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.450845 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.450850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:54.451019 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.451041 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 00:58:54.451149 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:58:54.451264 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":694,"threads_started":1}
I20260929 00:58:54.452566 17906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba370a11c7914fd683e3b8b460088ba6" candidate_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" is_pre_election: true
I20260929 00:58:54.452961 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba370a11c7914fd683e3b8b460088ba6" candidate_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd11e32ce41439bb3c725959377c059" is_pre_election: true
I20260929 00:58:54.453080 17906 raft_consensus.cc:2454] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7e146d4ffa4f30839766b820a4cfc3 in term 0.
I20260929 00:58:54.453441 17639 raft_consensus.cc:2454] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7e146d4ffa4f30839766b820a4cfc3 in term 0.
I20260929 00:58:54.453774 17706 leader_election.cc:302] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc7e146d4ffa4f30839766b820a4cfc3, d0bd73cb1cd2432c9a184b9a3328f2d5; no voters:
I20260929 00:58:54.454211 17977 raft_consensus.cc:2781] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:58:54.454370 17977 raft_consensus.cc:482] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:58:54.454505 17977 raft_consensus.cc:3037] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:54.456754 17977 raft_consensus.cc:504] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.457350 17977 leader_election.cc:288] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [CANDIDATE]: Term 1 election: Requested vote from peers d0bd73cb1cd2432c9a184b9a3328f2d5 (127.0.97.3:45469), 0dd11e32ce41439bb3c725959377c059 (127.0.97.1:38255)
I20260929 00:58:54.457710 17906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba370a11c7914fd683e3b8b460088ba6" candidate_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5"
I20260929 00:58:54.457815 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba370a11c7914fd683e3b8b460088ba6" candidate_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd11e32ce41439bb3c725959377c059"
I20260929 00:58:54.457904 17906 raft_consensus.cc:3037] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:54.458036 17639 raft_consensus.cc:3037] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:54.460125 17906 raft_consensus.cc:2454] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7e146d4ffa4f30839766b820a4cfc3 in term 1.
I20260929 00:58:54.460290 17639 raft_consensus.cc:2454] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7e146d4ffa4f30839766b820a4cfc3 in term 1.
I20260929 00:58:54.460546 17706 leader_election.cc:302] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc7e146d4ffa4f30839766b820a4cfc3, d0bd73cb1cd2432c9a184b9a3328f2d5; no voters:
I20260929 00:58:54.460816 17977 raft_consensus.cc:2781] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:54.461352 17977 raft_consensus.cc:686] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 1 LEADER]: Becoming Leader. State: Replica: bc7e146d4ffa4f30839766b820a4cfc3, State: Running, Role: LEADER
I20260929 00:58:54.462059 17977 consensus_queue.cc:237] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } }
I20260929 00:58:54.466351 17495 catalog_manager.cc:5495] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc7e146d4ffa4f30839766b820a4cfc3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e146d4ffa4f30839766b820a4cfc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 } health_report { overall_health: UNKNOWN } } }
W20260929 00:58:54.470366 17685 tablet.cc:2369] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:58:54.475490 17819 tablet.cc:2369] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:58:54.501612 17952 tablet.cc:2369] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:54.503249 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.498038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.97.2:40085)
0929 00:58:54.498279 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.498284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.498338 (+ 54us) server_negotiation.cc:407] Connection header received
0929 00:58:54.498520 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.498524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.498600 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.498724 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:58:54.499049 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.499764 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.501198 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.501581 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.501716 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.502178 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:58:54.502701 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:58:54.502857 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.502912 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.502988 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 00:58:54.511580 17639 raft_consensus.cc:1261] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 1 FOLLOWER]: Refusing update from remote peer bc7e146d4ffa4f30839766b820a4cfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:54.511579 17906 raft_consensus.cc:1261] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 1 FOLLOWER]: Refusing update from remote peer bc7e146d4ffa4f30839766b820a4cfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:54.512696 17981 consensus_queue.cc:1035] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd11e32ce41439bb3c725959377c059" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:54.513077 17977 consensus_queue.cc:1035] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0bd73cb1cd2432c9a184b9a3328f2d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:54.524531 17984 mvcc.cc:205] Tried to move back new op lower bound from 7334475917348573184 to 7334475917160456192. Current Snapshot: MvccSnapshot[applied={T|T < 7334475917348573184}]
I20260929 00:58:54.531834 17985 mvcc.cc:205] Tried to move back new op lower bound from 7334475917348573184 to 7334475917160456192. Current Snapshot: MvccSnapshot[applied={T|T < 7334475917348573184}]
I20260929 00:58:54.532624 17986 mvcc.cc:205] Tried to move back new op lower bound from 7334475917348573184 to 7334475917160456192. Current Snapshot: MvccSnapshot[applied={T|T < 7334475917348573184}]
I20260929 00:58:54.537607 17494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:54.537930 17494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:54.542660 17494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2bebf6fc3048ca9a306efe74166767: Sending DeleteTablet for 3 replicas of tablet ba370a11c7914fd683e3b8b460088ba6
I20260929 00:58:54.543715 17886 tablet_service.cc:1514] Processing DeleteTablet for tablet ba370a11c7914fd683e3b8b460088ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56222
I20260929 00:58:54.543722 17753 tablet_service.cc:1514] Processing DeleteTablet for tablet ba370a11c7914fd683e3b8b460088ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53372
I20260929 00:58:54.544026 17619 tablet_service.cc:1514] Processing DeleteTablet for tablet ba370a11c7914fd683e3b8b460088ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260929 00:58:54.546344 17885 tablet_service.cc:1514] Processing DeleteTablet for tablet ba370a11c7914fd683e3b8b460088ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56222
I20260929 00:58:54.546876 17618 tablet_service.cc:1514] Processing DeleteTablet for tablet ba370a11c7914fd683e3b8b460088ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260929 00:58:54.549310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17556
I20260929 00:58:54.556252 17996 tablet_replica.cc:330] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: stopping tablet replica
I20260929 00:58:54.556841 17996 raft_consensus.cc:2229] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:54.557468 17996 raft_consensus.cc:2258] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:54.558429 17995 tablet_replica.cc:330] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: stopping tablet replica
I20260929 00:58:54.558871 17995 raft_consensus.cc:2229] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:58:54.559515 17995 raft_consensus.cc:2258] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:54.560293 17994 tablet_replica.cc:330] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: stopping tablet replica
I20260929 00:58:54.560756 17994 raft_consensus.cc:2229] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:54.561347 17995 ts_tablet_manager.cc:1905] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:54.561375 17994 raft_consensus.cc:2258] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:54.563383 17994 ts_tablet_manager.cc:1905] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:54.566865 17995 ts_tablet_manager.cc:1918] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:54.567070 17995 log.cc:1196] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/ba370a11c7914fd683e3b8b460088ba6
I20260929 00:58:54.567634 17995 ts_tablet_manager.cc:1939] T ba370a11c7914fd683e3b8b460088ba6 P bc7e146d4ffa4f30839766b820a4cfc3: Deleting consensus metadata
I20260929 00:58:54.569613 17481 catalog_manager.cc:4841] TS bc7e146d4ffa4f30839766b820a4cfc3 (127.0.97.2:40085): tablet ba370a11c7914fd683e3b8b460088ba6 (table test-table [id=917ed36133464ee3a294b810cf1b87df]) successfully deleted
I20260929 00:58:54.570568 17994 ts_tablet_manager.cc:1918] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:54.570735 17994 log.cc:1196] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/ba370a11c7914fd683e3b8b460088ba6
I20260929 00:58:54.571274 17994 ts_tablet_manager.cc:1939] T ba370a11c7914fd683e3b8b460088ba6 P d0bd73cb1cd2432c9a184b9a3328f2d5: Deleting consensus metadata
I20260929 00:58:54.571928 17996 ts_tablet_manager.cc:1905] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:54.573006 17479 catalog_manager.cc:4841] TS d0bd73cb1cd2432c9a184b9a3328f2d5 (127.0.97.3:45469): tablet ba370a11c7914fd683e3b8b460088ba6 (table test-table [id=917ed36133464ee3a294b810cf1b87df]) successfully deleted
W20260929 00:58:54.574009 17479 catalog_manager.cc:4808] TS d0bd73cb1cd2432c9a184b9a3328f2d5 (127.0.97.3:45469): delete failed for tablet ba370a11c7914fd683e3b8b460088ba6 because the tablet was not found. No further retry: Not found: Tablet not found: ba370a11c7914fd683e3b8b460088ba6
Sep 29 00:58:54 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:58:54.578293 17996 ts_tablet_manager.cc:1918] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:54.578476 17996 log.cc:1196] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/ba370a11c7914fd683e3b8b460088ba6
I20260929 00:58:54.579063 17996 ts_tablet_manager.cc:1939] T ba370a11c7914fd683e3b8b460088ba6 P 0dd11e32ce41439bb3c725959377c059: Deleting consensus metadata
I20260929 00:58:54.581061 17480 catalog_manager.cc:4841] TS 0dd11e32ce41439bb3c725959377c059 (127.0.97.1:38255): tablet ba370a11c7914fd683e3b8b460088ba6 (table test-table [id=917ed36133464ee3a294b810cf1b87df]) successfully deleted
W20260929 00:58:54.581496 17480 catalog_manager.cc:4808] TS 0dd11e32ce41439bb3c725959377c059 (127.0.97.1:38255): delete failed for tablet ba370a11c7914fd683e3b8b460088ba6 because the tablet was not found. No further retry: Not found: Tablet not found: ba370a11c7914fd683e3b8b460088ba6
I20260929 00:58:54.584964 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.551303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.0.97.1:38255)
0929 00:58:54.551528 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.551537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.566186 (+ 14649us) server_negotiation.cc:407] Connection header received
0929 00:58:54.566395 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.566400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.566510 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.566631 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:54.568810 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.569790 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.571616 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.572133 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.572265 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.577769 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:54.577783 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:54.577801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:54.577842 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:54.581929 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.582808 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.582814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.582817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.582942 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.583331 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.583334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.583336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.583662 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:54.584031 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.584293 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.584534 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:58:54.585664 17679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:54.676179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17556
I20260929 00:58:54.694532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17690
Sep 29 00:58:54 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:58:54.712175 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.695329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.97.2:40085)
0929 00:58:54.695524 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.695528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.695549 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:58:54.695741 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.695745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.695807 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.695887 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:54.697525 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.698325 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.700197 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.700763 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.700857 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.706139 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:54.706153 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:54.706169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:54.706212 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:54.709493 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.710277 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.710283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.710286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.710434 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.710848 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.710851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.710853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.711166 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:54.711404 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.711726 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.711876 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:58:54.713083 17813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:54.824429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17690
I20260929 00:58:54.842640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17823
Sep 29 00:58:54 dist-test-slave-pzsx krb5kdc[17450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:58:54.857542 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:54.843407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.97.3:45469)
0929 00:58:54.843586 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:54.843590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:54.843608 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:58:54.843746 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:54.843750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:54.843821 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:54.843906 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:54.845258 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.846023 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.847351 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:54.847747 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:54.847888 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:54.852143 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:54.852158 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:54.852174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:54.852206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:54.855013 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.855578 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.855585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.855587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.855705 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:54.856011 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:54.856014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:54.856016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:54.856330 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:54.856654 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:54.857047 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 00:58:54.857283 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:58:54.858245 17946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:54.946463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17823
I20260929 00:58:54.964830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 17463
I20260929 00:58:54.965850 17525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:55.076632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 17463
2026-09-29T00:58:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4215 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T00:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:58:55Z Disabled control of system clock
I20260929 00:58:55.155131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:58:55.482563 18005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:55.482836 18005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:55.482914 18005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:55.490039 18005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:55.490137 18005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:55.490180 18005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:55.490237 18005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:55.490283 18005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:55.499187 18005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:55.500350 18005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:55.502080 18005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:55.513850 18013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:55.513942 18010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:55.513921 18011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:55.514865 18005 server_base.cc:1034] running on GCE node
I20260929 00:58:55.515647 18005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:55.517163 18005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:55.518385 18005 hybrid_clock.cc:648] HybridClock initialized: now 1790643535518328 us; error 55 us; skew 500 ppm
I20260929 00:58:55.518754 18005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:58:55.524928 18005 webserver.cc:460] Webserver started at http://127.0.97.62:38497/ using document root <none> and password file <none>
I20260929 00:58:55.525636 18005 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:55.525766 18005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:55.526109 18005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:55.529105 18005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "d4e9534470df44c0b9e5b4f49eb79faa"
format_stamp: "Formatted at 2026-09-29 00:58:55 on dist-test-slave-pzsx"
server_key: "fe1d1eb49a8729a20da4f2988f619be3"
server_key_iv: "ce8b0aae28e45321c8f37ed3687cd667"
server_key_version: "encryptionkey@0"
I20260929 00:58:55.529928 18005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "d4e9534470df44c0b9e5b4f49eb79faa"
format_stamp: "Formatted at 2026-09-29 00:58:55 on dist-test-slave-pzsx"
server_key: "fe1d1eb49a8729a20da4f2988f619be3"
server_key_iv: "ce8b0aae28e45321c8f37ed3687cd667"
server_key_version: "encryptionkey@0"
I20260929 00:58:55.535085 18005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:58:55.538719 18019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:55.540144 18005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:58:55.540343 18005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "d4e9534470df44c0b9e5b4f49eb79faa"
format_stamp: "Formatted at 2026-09-29 00:58:55 on dist-test-slave-pzsx"
server_key: "fe1d1eb49a8729a20da4f2988f619be3"
server_key_iv: "ce8b0aae28e45321c8f37ed3687cd667"
server_key_version: "encryptionkey@0"
I20260929 00:58:55.540601 18005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:55.567447 18005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:55.568372 18005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:55.568710 18005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:55.587024 18005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40383
I20260929 00:58:55.587044 18070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40383 every 8 connection(s)
I20260929 00:58:55.588452 18005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:55.592094 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:55.598256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18005
I20260929 00:58:55.598428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:58:55.598870 388 external_mini_cluster.cc:1428] Setting key d437349eb0ad0388278ed8b2a54bb1c9
I20260929 00:58:55.602900 18071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: Bootstrap starting.
I20260929 00:58:55.606345 18071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:55.607659 18071 log.cc:824] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:55.610997 18071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: No bootstrap required, opened a new log
I20260929 00:58:55.613202 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:55.600911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.0.97.62:40383)
0929 00:58:55.601905 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:55.601916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:55.601958 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:58:55.603738 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:55.603761 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:55.604230 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:55.604632 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:55.605241 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:55.606548 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:55.608557 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:55.609411 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:55.609554 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:55.610773 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:55.610785 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:55.610805 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:55.611433 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:55.611522 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:55.611544 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:55.611628 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:55.611868 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 00:58:55.612189 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":371,"threads_started":1}
I20260929 00:58:55.618774 18071 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 00:58:55.619153 18071 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:55.619277 18071 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e9534470df44c0b9e5b4f49eb79faa, State: Initialized, Role: FOLLOWER
I20260929 00:58:55.619875 18071 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 00:58:55.620118 18071 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:55.620275 18071 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:55.620450 18071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:55.622816 18071 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 00:58:55.623327 18071 leader_election.cc:302] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e9534470df44c0b9e5b4f49eb79faa; no voters:
I20260929 00:58:55.623924 18071 leader_election.cc:288] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:58:55.624082 18076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:55.625623 18076 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [term 1 LEADER]: Becoming Leader. State: Replica: d4e9534470df44c0b9e5b4f49eb79faa, State: Running, Role: LEADER
I20260929 00:58:55.626284 18071 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:55.626183 18076 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 00:58:55.629151 18078 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e9534470df44c0b9e5b4f49eb79faa. Latest consensus state: current_term: 1 leader_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260929 00:58:55.629565 18078 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:55.630101 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e9534470df44c0b9e5b4f49eb79faa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260929 00:58:55.630407 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:55.631083 18085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:55.636476 18085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:55.647039 18085 catalog_manager.cc:1347] Generated new cluster ID: 21184710eca74e2abc26f645a54c485e
I20260929 00:58:55.647199 18085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:55.687407 18085 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:58:55.689337 18085 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:55.699826 18085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: Generated new TSK 0
I20260929 00:58:55.700851 18085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:58:55.717852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:58:56.051576 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:56.051824 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:56.051910 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:56.058887 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:56.058979 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:56.059082 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:58:56.067852 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:56.069020 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:56.070784 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:56.083273 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.083355 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.083312 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:56.084090 18095 server_base.cc:1034] running on GCE node
I20260929 00:58:56.084714 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:56.085666 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:56.086866 18095 hybrid_clock.cc:648] HybridClock initialized: now 1790643536086827 us; error 31 us; skew 500 ppm
I20260929 00:58:56.087175 18095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:58:56.089422 18095 webserver.cc:460] Webserver started at http://127.0.97.1:45231/ using document root <none> and password file <none>
I20260929 00:58:56.090095 18095 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:56.090201 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:56.090513 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:56.093250 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "a03f29944b3c4c999e09286a4299b789"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c9295a919e20f22b5030895b0cc2b4c6"
server_key_iv: "2894e286cd4ce06a6226e846fdeb71a8"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.093997 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a03f29944b3c4c999e09286a4299b789"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c9295a919e20f22b5030895b0cc2b4c6"
server_key_iv: "2894e286cd4ce06a6226e846fdeb71a8"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.098795 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 00:58:56.102164 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.103428 18095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:58:56.103605 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "a03f29944b3c4c999e09286a4299b789"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c9295a919e20f22b5030895b0cc2b4c6"
server_key_iv: "2894e286cd4ce06a6226e846fdeb71a8"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.103832 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:56.119081 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:56.119827 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:56.120086 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:56.121037 18095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:56.122421 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:56.122565 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.122699 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:56.122780 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.156996 18095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38585
I20260929 00:58:56.157049 18221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38585 every 8 connection(s)
I20260929 00:58:56.158341 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:58:56.162113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18095
I20260929 00:58:56.162283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:58:56.162853 388 external_mini_cluster.cc:1428] Setting key e30370bbb40ad8017a1aa37126e89eec
I20260929 00:58:56.165903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:58:56.172576 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:56.160891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59915 (local address 127.0.97.62:40383)
0929 00:58:56.161359 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:56.161364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:56.162745 (+ 1381us) server_negotiation.cc:407] Connection header received
0929 00:58:56.164981 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:56.164985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:56.165045 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:56.165135 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:56.166545 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.167278 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.169755 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.170129 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.170228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:56.171831 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:56.171835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:56.171838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:56.171857 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:56.171898 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:56.171909 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:56.171974 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:56.172218 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 00:58:56.172288 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":269,"threads_started":1}
I20260929 00:58:56.173949 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:56.161411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:59915)
0929 00:58:56.162556 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:56.162611 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:56.164685 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:56.165405 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:56.165417 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:56.165534 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:56.166380 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:56.166406 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.167438 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.167446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:56.169620 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:56.169638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.170353 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.170362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:56.170681 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:56.171468 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:58:56.171473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:56.171525 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:58:56.171640 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:56.172006 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:56.172019 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:56.172029 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:58:56.172202 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 00:58:56.172618 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260929 00:58:56.175494 18222 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260929 00:58:56.175784 18222 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:56.176707 18222 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260929 00:58:56.178787 18036 ts_manager.cc:194] Registered new tserver with Master: a03f29944b3c4c999e09286a4299b789 (127.0.97.1:38585)
I20260929 00:58:56.180253 18036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59915
W20260929 00:58:56.491039 18226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:56.491314 18226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:56.491474 18226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:56.498543 18226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:56.498647 18226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:56.498811 18226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:58:56.507963 18226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:56.509316 18226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:56.511085 18226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:56.523927 18234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.523954 18231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.524279 18232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:56.525307 18226 server_base.cc:1034] running on GCE node
I20260929 00:58:56.525990 18226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:56.527006 18226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:56.528234 18226 hybrid_clock.cc:648] HybridClock initialized: now 1790643536528192 us; error 34 us; skew 500 ppm
I20260929 00:58:56.528626 18226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:58:56.530997 18226 webserver.cc:460] Webserver started at http://127.0.97.2:37849/ using document root <none> and password file <none>
I20260929 00:58:56.531697 18226 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:56.531826 18226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:56.532153 18226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:56.535221 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f5ceeeaa37946cba92a962e394dd82f"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "56b8b6adf746bf9afb3a0f6644c9948c"
server_key_iv: "80504bebfbabe01a0ef83db1cdb6248b"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.536028 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f5ceeeaa37946cba92a962e394dd82f"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "56b8b6adf746bf9afb3a0f6644c9948c"
server_key_iv: "80504bebfbabe01a0ef83db1cdb6248b"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.541097 18226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:58:56.544705 18240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.546059 18226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:58:56.546253 18226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "2f5ceeeaa37946cba92a962e394dd82f"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "56b8b6adf746bf9afb3a0f6644c9948c"
server_key_iv: "80504bebfbabe01a0ef83db1cdb6248b"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.546480 18226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:56.567035 18226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:56.567883 18226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:56.568173 18226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:56.569199 18226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:56.570624 18226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:56.570742 18226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.570879 18226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:56.570968 18226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.607139 18226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42503
I20260929 00:58:56.607203 18352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42503 every 8 connection(s)
I20260929 00:58:56.608620 18226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:58:56.612241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18226
I20260929 00:58:56.612411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:58:56.613150 388 external_mini_cluster.cc:1428] Setting key 7c929c87dd6c95b0d110254c6ee3bea6
I20260929 00:58:56.615784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:58:56.621119 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:56.611073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57757 (local address 127.0.97.62:40383)
0929 00:58:56.611245 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:56.611249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:56.612717 (+ 1468us) server_negotiation.cc:407] Connection header received
0929 00:58:56.614746 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:56.614752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:56.614843 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:56.614947 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:56.616113 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.616902 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.618434 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.618818 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.618950 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:56.620306 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:56.620311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:56.620314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:56.620334 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:56.620365 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:56.620377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:56.620469 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:56.620731 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 00:58:56.620824 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:58:56.622190 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:56.611545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:57757)
0929 00:58:56.612474 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:56.612576 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:56.614470 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:56.615175 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:56.615184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:56.615279 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:56.615877 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:56.615901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.617051 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.617057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:56.618282 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:56.618294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:56.619034 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:56.619041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:56.619266 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:56.619989 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:58:56.619991 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:56.620030 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:58:56.620111 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:56.620470 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:56.620483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:56.620494 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 00:58:56.620710 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 00:58:56.621096 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":279,"threads_started":1}
I20260929 00:58:56.623646 18353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260929 00:58:56.623950 18353 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:56.624738 18353 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260929 00:58:56.626233 18036 ts_manager.cc:194] Registered new tserver with Master: 2f5ceeeaa37946cba92a962e394dd82f (127.0.97.2:42503)
I20260929 00:58:56.627455 18036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:57757
W20260929 00:58:56.938900 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:56.939160 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:56.939246 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:56.946480 18356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:56.946573 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:56.946677 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:58:56.955590 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:56.956854 18356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:56.958653 18356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:56.971405 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.971405 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:56.971483 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:56.972834 18356 server_base.cc:1034] running on GCE node
I20260929 00:58:56.973402 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:56.974292 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:56.975551 18356 hybrid_clock.cc:648] HybridClock initialized: now 1790643536975503 us; error 38 us; skew 500 ppm
I20260929 00:58:56.975894 18356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:58:56.978101 18356 webserver.cc:460] Webserver started at http://127.0.97.3:39113/ using document root <none> and password file <none>
I20260929 00:58:56.978785 18356 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:56.978914 18356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:56.979249 18356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:56.982050 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "0db2bbbec2804a35babe19ad44e063aa"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c4af29d67c75aacb7c6c20240b3ec87b"
server_key_iv: "82bd7ece8bca460a5e63bc6115b3db39"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.982817 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0db2bbbec2804a35babe19ad44e063aa"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c4af29d67c75aacb7c6c20240b3ec87b"
server_key_iv: "82bd7ece8bca460a5e63bc6115b3db39"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.987689 18356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 00:58:56.990962 18370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:56.992161 18356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:58:56.992331 18356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "0db2bbbec2804a35babe19ad44e063aa"
format_stamp: "Formatted at 2026-09-29 00:58:56 on dist-test-slave-pzsx"
server_key: "c4af29d67c75aacb7c6c20240b3ec87b"
server_key_iv: "82bd7ece8bca460a5e63bc6115b3db39"
server_key_version: "encryptionkey@0"
I20260929 00:58:56.992587 18356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:57.003197 18356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:57.003932 18356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:57.004202 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:57.005151 18356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:58:57.006601 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:58:57.006723 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:57.006847 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:58:57.006939 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:57.039674 18356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35579
I20260929 00:58:57.039721 18482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35579 every 8 connection(s)
I20260929 00:58:57.041039 18356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:58:57.050611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18356
I20260929 00:58:57.050798 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:58:57.051364 388 external_mini_cluster.cc:1428] Setting key ee8503fc565f80e156460a0e2114e251
I20260929 00:58:57.054085 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.043543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40899 (local address 127.0.97.62:40383)
0929 00:58:57.043722 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.043727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.045249 (+ 1522us) server_negotiation.cc:407] Connection header received
0929 00:58:57.047005 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.047011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.047101 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.047220 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:57.048242 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.049033 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.051262 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.051793 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.051936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.053271 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:57.053275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:57.053279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:57.053297 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:57.053326 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:57.053337 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:57.053435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.053657 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.053754 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:58:57.055080 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.044134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:40899)
0929 00:58:57.045092 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.045136 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.046747 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.047446 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.047454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.047537 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:58:57.048076 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.048091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.049184 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.049196 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.051115 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.051130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.052022 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.052027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.052220 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.052916 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:58:57.052922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:58:57.052976 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:58:57.053065 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:58:57.053432 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:58:57.053444 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:58:57.053454 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.053648 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.054028 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":357,"threads_started":1}
I20260929 00:58:57.056464 18483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260929 00:58:57.056775 18483 heartbeater.cc:464] Registering TS with master...
I20260929 00:58:57.057462 18483 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260929 00:58:57.058761 18036 ts_manager.cc:194] Registered new tserver with Master: 0db2bbbec2804a35babe19ad44e063aa (127.0.97.3:35579)
I20260929 00:58:57.059517 18036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40899
I20260929 00:58:57.066277 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:58:57.075488 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.070158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.0.97.62:40383)
0929 00:58:57.070411 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.070420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.070728 (+ 308us) server_negotiation.cc:407] Connection header received
0929 00:58:57.070855 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.070859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.070933 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.071012 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:57.071417 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.072076 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.073478 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.073881 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.073970 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.074857 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:57.074861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:57.074864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:57.074881 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:57.074909 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:57.074920 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:57.074986 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.075102 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.075169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:58:57.079167 18036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:58:57.085824 18036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:58:57.107412 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.097205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35579 (local address 127.0.0.1:57812)
0929 00:58:57.098615 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.098653 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.098846 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.099589 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.099596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.099636 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.100024 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.100036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.102276 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.102284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.104932 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.104946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.106316 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.106322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.106531 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.106589 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.106754 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.106993 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"spinlock_wait_cycles":6656,"thread_start_us":1076,"threads_started":1}
I20260929 00:58:57.107372 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.097520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.97.3:35579)
0929 00:58:57.098323 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.098334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.098762 (+ 428us) server_negotiation.cc:407] Connection header received
0929 00:58:57.099062 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.099068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.099346 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.100228 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.100480 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.102129 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.105169 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.106116 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.106270 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.106551 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.106797 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.106998 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":458,"threads_started":1}
I20260929 00:58:57.108152 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.098959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60456)
0929 00:58:57.099513 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.099532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.099637 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.100749 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.100755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.100780 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.101180 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.101192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.102879 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.102887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.105146 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.105159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.107266 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.107272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.107480 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.107568 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.107648 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.107769 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":44,"spinlock_wait_cycles":7552,"thread_start_us":266,"threads_started":1}
I20260929 00:58:57.108695 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.099221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.0.97.1:38585)
0929 00:58:57.100120 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.100129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.100167 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:58:57.100262 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.100271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.100529 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.101361 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.101557 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.102748 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.105341 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.106749 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.106895 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.107142 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.108125 (+ 983us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.108294 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":639,"threads_started":1}
I20260929 00:58:57.111573 18418 tablet_service.cc:1467] Processing CreateTablet for tablet bb95a626ecd14a869607207214a669b6 (DEFAULT_TABLE table=test-table [id=52a9af633c28408fb03979caf87d75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:57.111830 18157 tablet_service.cc:1467] Processing CreateTablet for tablet bb95a626ecd14a869607207214a669b6 (DEFAULT_TABLE table=test-table [id=52a9af633c28408fb03979caf87d75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:57.113142 18418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb95a626ecd14a869607207214a669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:57.113183 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.100173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.0.97.2:42503)
0929 00:58:57.103308 (+ 3135us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.103318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.104018 (+ 700us) server_negotiation.cc:407] Connection header received
0929 00:58:57.104124 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.104131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.104354 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.104540 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.105298 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.106827 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.110584 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.111963 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.112107 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.112376 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.112628 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.112799 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2957,"thread_start_us":2833,"threads_started":1}
I20260929 00:58:57.113211 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb95a626ecd14a869607207214a669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:57.113884 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.099563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42503 (local address 127.0.0.1:41864)
0929 00:58:57.103759 (+ 4196us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.103781 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.103897 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.104611 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.104616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.104644 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.105118 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.105132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.106984 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.106989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.110382 (+ 3393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.110397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.112169 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.112175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.112345 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.112377 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.112457 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.113610 (+ 1153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4067,"thread_start_us":3892,"threads_started":1}
I20260929 00:58:57.117547 18288 tablet_service.cc:1467] Processing CreateTablet for tablet bb95a626ecd14a869607207214a669b6 (DEFAULT_TABLE table=test-table [id=52a9af633c28408fb03979caf87d75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:58:57.118975 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb95a626ecd14a869607207214a669b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:57.124874 18501 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Bootstrap starting.
I20260929 00:58:57.127728 18502 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Bootstrap starting.
I20260929 00:58:57.129014 18501 tablet_bootstrap.cc:654] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:57.130338 18501 log.cc:824] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:57.131439 18502 tablet_bootstrap.cc:654] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:57.133033 18502 log.cc:824] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:57.133667 18504 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Bootstrap starting.
I20260929 00:58:57.136241 18501 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: No bootstrap required, opened a new log
I20260929 00:58:57.136579 18501 ts_tablet_manager.cc:1397] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 00:58:57.136864 18502 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: No bootstrap required, opened a new log
I20260929 00:58:57.137140 18502 ts_tablet_manager.cc:1397] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 00:58:57.137187 18504 tablet_bootstrap.cc:654] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:57.138316 18504 log.cc:824] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:57.141166 18504 tablet_bootstrap.cc:492] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: No bootstrap required, opened a new log
I20260929 00:58:57.141418 18504 ts_tablet_manager.cc:1397] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 00:58:57.144019 18501 raft_consensus.cc:348] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.144529 18501 raft_consensus.cc:374] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:57.144642 18501 raft_consensus.cc:729] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03f29944b3c4c999e09286a4299b789, State: Initialized, Role: FOLLOWER
I20260929 00:58:57.145377 18501 consensus_queue.cc:260] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.147388 18502 raft_consensus.cc:348] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.147801 18502 raft_consensus.cc:374] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:57.147907 18502 raft_consensus.cc:729] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db2bbbec2804a35babe19ad44e063aa, State: Initialized, Role: FOLLOWER
I20260929 00:58:57.148643 18222 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260929 00:58:57.148614 18502 consensus_queue.cc:260] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.149641 18501 ts_tablet_manager.cc:1428] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Time spent starting tablet: real 0.013s user 0.002s sys 0.011s
I20260929 00:58:57.150580 18483 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260929 00:58:57.151407 18502 ts_tablet_manager.cc:1428] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260929 00:58:57.152141 18504 raft_consensus.cc:348] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.152491 18504 raft_consensus.cc:374] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:57.152661 18504 raft_consensus.cc:729] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5ceeeaa37946cba92a962e394dd82f, State: Initialized, Role: FOLLOWER
I20260929 00:58:57.153267 18504 consensus_queue.cc:260] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.154395 18353 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260929 00:58:57.155020 18504 ts_tablet_manager.cc:1428] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 00:58:57.160598 18223 tablet.cc:2369] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:58:57.292744 18484 tablet.cc:2369] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:57.352187 18507 raft_consensus.cc:482] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:58:57.352567 18507 raft_consensus.cc:504] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.354388 18507 leader_election.cc:288] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db2bbbec2804a35babe19ad44e063aa (127.0.97.3:35579), 2f5ceeeaa37946cba92a962e394dd82f (127.0.97.2:42503)
I20260929 00:58:57.359988 18509 raft_consensus.cc:482] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 00:58:57.360663 18354 tablet.cc:2369] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:58:57.360378 18509 raft_consensus.cc:504] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.361907 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.354878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39171 (local address 127.0.97.3:35579)
0929 00:58:57.355083 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.355088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.355637 (+ 549us) server_negotiation.cc:407] Connection header received
0929 00:58:57.355847 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.355854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.355929 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.356056 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.356903 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.357621 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.359501 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.360766 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.360890 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.361005 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.361515 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.361653 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:58:57.362316 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.355934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43921 (local address 127.0.97.2:42503)
0929 00:58:57.356193 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.356199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.356670 (+ 471us) server_negotiation.cc:407] Connection header received
0929 00:58:57.356849 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.356856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.356919 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.357023 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.357549 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.358240 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.359903 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.361278 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.361404 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.361516 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.361955 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.362094 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:58:57.362460 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.354780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35579 (local address 127.0.97.1:39171)
0929 00:58:57.355515 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.355536 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.355687 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.356277 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.356281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.356314 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.356760 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.356771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.357762 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.357768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.359274 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.359286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.361122 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.361130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.361300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.361352 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.362085 (+ 733us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.362191 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":417,"threads_started":1}
I20260929 00:58:57.362617 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.355664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42503 (local address 127.0.97.1:43921)
0929 00:58:57.356470 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.356563 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.356701 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.357039 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.357043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.357065 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.357376 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.357384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.358373 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.358379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.359734 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.359744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.361623 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.361629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.361784 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.361812 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.361906 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.362132 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":490,"threads_started":1}
I20260929 00:58:57.362895 18509 leader_election.cc:288] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db2bbbec2804a35babe19ad44e063aa (127.0.97.3:35579), a03f29944b3c4c999e09286a4299b789 (127.0.97.1:38585)
I20260929 00:58:57.363646 18438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "a03f29944b3c4c999e09286a4299b789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2bbbec2804a35babe19ad44e063aa" is_pre_election: true
I20260929 00:58:57.364018 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "a03f29944b3c4c999e09286a4299b789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5ceeeaa37946cba92a962e394dd82f" is_pre_election: true
I20260929 00:58:57.364216 18438 raft_consensus.cc:2454] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03f29944b3c4c999e09286a4299b789 in term 0.
I20260929 00:58:57.364465 18308 raft_consensus.cc:2454] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03f29944b3c4c999e09286a4299b789 in term 0.
I20260929 00:58:57.365242 18113 leader_election.cc:302] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5ceeeaa37946cba92a962e394dd82f, a03f29944b3c4c999e09286a4299b789; no voters:
I20260929 00:58:57.365728 18507 raft_consensus.cc:2781] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:58:57.365890 18507 raft_consensus.cc:482] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:58:57.365994 18507 raft_consensus.cc:3037] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:57.369022 18507 raft_consensus.cc:504] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.369727 18507 leader_election.cc:288] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [CANDIDATE]: Term 1 election: Requested vote from peers 0db2bbbec2804a35babe19ad44e063aa (127.0.97.3:35579), 2f5ceeeaa37946cba92a962e394dd82f (127.0.97.2:42503)
I20260929 00:58:57.370218 18438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "a03f29944b3c4c999e09286a4299b789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2bbbec2804a35babe19ad44e063aa"
I20260929 00:58:57.370362 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "a03f29944b3c4c999e09286a4299b789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5ceeeaa37946cba92a962e394dd82f"
I20260929 00:58:57.370417 18438 raft_consensus.cc:3037] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:57.370558 18308 raft_consensus.cc:3037] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:57.370970 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.363131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53679 (local address 127.0.97.3:35579)
0929 00:58:57.363341 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.363350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.363965 (+ 615us) server_negotiation.cc:407] Connection header received
0929 00:58:57.364234 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.364241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.364337 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.364419 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.365364 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.366480 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.368566 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.369780 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.369904 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.370007 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.370538 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.370666 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:58:57.371651 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.363071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35579 (local address 127.0.97.2:53679)
0929 00:58:57.363869 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.363889 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.364063 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.364664 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.364672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.364706 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.365185 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.365203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.366616 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.366627 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.368348 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.368365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.370158 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.370165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.370343 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.370380 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.371169 (+ 789us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.371321 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":487,"threads_started":1}
I20260929 00:58:57.372431 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.364635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52839 (local address 127.0.97.1:38585)
0929 00:58:57.364802 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.364808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.365736 (+ 928us) server_negotiation.cc:407] Connection header received
0929 00:58:57.365958 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.365963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.366076 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.366146 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:58:57.367002 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.368070 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.370155 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.371374 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.371467 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.371549 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.372077 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.372201 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:58:57.372593 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.364005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.97.2:52839)
0929 00:58:57.365647 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.365667 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:58:57.365822 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:58:57.366335 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:58:57.366339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:58:57.366368 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:58:57.366828 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.366843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.368232 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.368238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.369937 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:58:57.369951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.371723 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.371729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:58:57.371914 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.371948 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:58:57.372041 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:58:57.372163 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"mutex_wait_us":2,"thread_start_us":1297,"threads_started":1}
I20260929 00:58:57.373268 18438 raft_consensus.cc:2454] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03f29944b3c4c999e09286a4299b789 in term 1.
I20260929 00:58:57.373524 18308 raft_consensus.cc:2454] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03f29944b3c4c999e09286a4299b789 in term 1.
I20260929 00:58:57.374142 18113 leader_election.cc:302] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5ceeeaa37946cba92a962e394dd82f, a03f29944b3c4c999e09286a4299b789; no voters:
I20260929 00:58:57.374142 18438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "2f5ceeeaa37946cba92a962e394dd82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2bbbec2804a35babe19ad44e063aa" is_pre_election: true
I20260929 00:58:57.374440 18438 raft_consensus.cc:2379] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f5ceeeaa37946cba92a962e394dd82f in current term 1: Already voted for candidate a03f29944b3c4c999e09286a4299b789 in this term.
I20260929 00:58:57.374483 18507 raft_consensus.cc:2781] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:57.374444 18177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb95a626ecd14a869607207214a669b6" candidate_uuid: "2f5ceeeaa37946cba92a962e394dd82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03f29944b3c4c999e09286a4299b789" is_pre_election: true
I20260929 00:58:57.375242 18507 raft_consensus.cc:686] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 1 LEADER]: Becoming Leader. State: Replica: a03f29944b3c4c999e09286a4299b789, State: Running, Role: LEADER
I20260929 00:58:57.375689 18507 consensus_queue.cc:237] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } }
I20260929 00:58:57.379400 18244 leader_election.cc:302] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f5ceeeaa37946cba92a962e394dd82f; no voters: 0db2bbbec2804a35babe19ad44e063aa, a03f29944b3c4c999e09286a4299b789
I20260929 00:58:57.380074 18509 raft_consensus.cc:2726] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 00:58:57.380888 18035 catalog_manager.cc:5495] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 reported cstate change: term changed from 0 to 1, leader changed from <none> to a03f29944b3c4c999e09286a4299b789 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a03f29944b3c4c999e09286a4299b789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03f29944b3c4c999e09286a4299b789" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 } health_report { overall_health: UNKNOWN } } }
I20260929 00:58:57.401432 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.395133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.97.1:38585)
0929 00:58:57.395325 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.395333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.395483 (+ 150us) server_negotiation.cc:407] Connection header received
0929 00:58:57.395734 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.395741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.395883 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.396081 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:58:57.397369 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.398132 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.399422 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.399834 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.399924 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.400413 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:58:57.400970 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:58:57.401094 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.401157 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.401219 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:58:57.409463 18438 raft_consensus.cc:1261] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 1 FOLLOWER]: Refusing update from remote peer a03f29944b3c4c999e09286a4299b789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:57.409461 18308 raft_consensus.cc:1261] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 1 FOLLOWER]: Refusing update from remote peer a03f29944b3c4c999e09286a4299b789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:58:57.410327 18514 consensus_queue.cc:1035] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db2bbbec2804a35babe19ad44e063aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:57.410694 18507 consensus_queue.cc:1035] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5ceeeaa37946cba92a962e394dd82f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:58:57.422626 18517 mvcc.cc:205] Tried to move back new op lower bound from 7334475929219051520 to 7334475929094500352. Current Snapshot: MvccSnapshot[applied={T|T < 7334475929219051520}]
I20260929 00:58:57.425274 18519 mvcc.cc:205] Tried to move back new op lower bound from 7334475929219051520 to 7334475929094500352. Current Snapshot: MvccSnapshot[applied={T|T < 7334475929219051520}]
I20260929 00:58:57.427192 18518 mvcc.cc:205] Tried to move back new op lower bound from 7334475929219051520 to 7334475929094500352. Current Snapshot: MvccSnapshot[applied={T|T < 7334475929219051520}]
I20260929 00:58:57.437098 18036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:57.437381 18036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:58:57.440846 18036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e9534470df44c0b9e5b4f49eb79faa: Sending DeleteTablet for 3 replicas of tablet bb95a626ecd14a869607207214a669b6
I20260929 00:58:57.441947 18418 tablet_service.cc:1514] Processing DeleteTablet for tablet bb95a626ecd14a869607207214a669b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:57 UTC) from {username='slave'} at 127.0.0.1:57812
I20260929 00:58:57.442112 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet bb95a626ecd14a869607207214a669b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:57 UTC) from {username='slave'} at 127.0.0.1:41864
I20260929 00:58:57.443034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18095
I20260929 00:58:57.443928 18157 tablet_service.cc:1514] Processing DeleteTablet for tablet bb95a626ecd14a869607207214a669b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:58:57 UTC) from {username='slave'} at 127.0.0.1:60456
I20260929 00:58:57.453915 18528 tablet_replica.cc:330] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: stopping tablet replica
I20260929 00:58:57.454698 18528 raft_consensus.cc:2229] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:57.455411 18528 raft_consensus.cc:2258] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:57.456110 18527 tablet_replica.cc:330] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: stopping tablet replica
I20260929 00:58:57.456918 18527 raft_consensus.cc:2229] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:58:57.457296 18529 tablet_replica.cc:330] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: stopping tablet replica
I20260929 00:58:57.457542 18527 raft_consensus.cc:2258] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:57.457686 18529 raft_consensus.cc:2229] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:58:57.458735 18528 ts_tablet_manager.cc:1905] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:57.459698 18527 ts_tablet_manager.cc:1905] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:57.465744 18528 ts_tablet_manager.cc:1918] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:57.465901 18528 log.cc:1196] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/bb95a626ecd14a869607207214a669b6
I20260929 00:58:57.466434 18527 ts_tablet_manager.cc:1918] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:58:57.466483 18528 ts_tablet_manager.cc:1939] T bb95a626ecd14a869607207214a669b6 P 2f5ceeeaa37946cba92a962e394dd82f: Deleting consensus metadata
I20260929 00:58:57.466607 18527 log.cc:1196] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/bb95a626ecd14a869607207214a669b6
I20260929 00:58:57.467123 18527 ts_tablet_manager.cc:1939] T bb95a626ecd14a869607207214a669b6 P 0db2bbbec2804a35babe19ad44e063aa: Deleting consensus metadata
I20260929 00:58:57.468492 18023 catalog_manager.cc:4841] TS 2f5ceeeaa37946cba92a962e394dd82f (127.0.97.2:42503): tablet bb95a626ecd14a869607207214a669b6 (table test-table [id=52a9af633c28408fb03979caf87d75da]) successfully deleted
I20260929 00:58:57.468851 18529 raft_consensus.cc:2258] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:58:57.468884 18023 catalog_manager.cc:4841] TS 0db2bbbec2804a35babe19ad44e063aa (127.0.97.3:35579): tablet bb95a626ecd14a869607207214a669b6 (table test-table [id=52a9af633c28408fb03979caf87d75da]) successfully deleted
I20260929 00:58:57.465068 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.446934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.97.1:38585)
0929 00:58:57.447368 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.447374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.456407 (+ 9033us) server_negotiation.cc:407] Connection header received
0929 00:58:57.456796 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.456803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.456969 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.457092 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:57.457801 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.458948 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.460943 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.461683 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.461788 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.463298 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:57.463309 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:57.463327 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:57.464135 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:57.464196 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:57.464227 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:57.464305 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.464597 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.464716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260929 00:58:57.469820 18217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:57.470752 18529 ts_tablet_manager.cc:1905] T bb95a626ecd14a869607207214a669b6 P a03f29944b3c4c999e09286a4299b789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:58:57.566752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18095
I20260929 00:58:57.585407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18226
W20260929 00:58:57.585460 18023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38585 (error 108)
I20260929 00:58:57.587693 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.587158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60488)
0929 00:58:57.587354 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.587539 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 00:58:57.587947 18023 catalog_manager.cc:4584] TS a03f29944b3c4c999e09286a4299b789 (127.0.97.1:38585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb95a626ecd14a869607207214a669b6: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
I20260929 00:58:57.593405 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.586399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.97.2:42503)
0929 00:58:57.586575 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.586579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.586595 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:58:57.586687 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.586693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.586789 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.586880 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:57.587437 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.588433 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.590280 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.590680 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.590770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.591869 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:57.591877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:57.591890 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:57.592460 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:57.592722 (+ 262us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:57.592745 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:57.592845 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.592984 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.593067 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:58:57.594033 18348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:57.641491 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.640947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60492)
0929 00:58:57.641184 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.641291 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 00:58:57.642591 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.642187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60504)
0929 00:58:57.642354 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.642438 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 00:58:57.689910 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18226
I20260929 00:58:57.703245 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.702777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60506)
0929 00:58:57.702999 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.703088 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 00:58:57.704334 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.704024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60520)
0929 00:58:57.704155 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.704224 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 00:58:57.708150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18356
I20260929 00:58:57.714579 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.708847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.97.3:35579)
0929 00:58:57.709033 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:57.709038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:57.709056 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:58:57.709219 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:57.709223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:57.709306 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:57.709392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:57.709780 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.710434 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.711633 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:57.712022 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:57.712154 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:57.713295 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:57.713304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:57.713320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:58:57.713894 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:57.713941 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:58:57.713959 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:57.714082 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:57.714186 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 00:58:57.714279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:58:57.715339 18478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:57.785557 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.785071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60534)
0929 00:58:57.785305 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.785390 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 00:58:57.786587 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:57.786210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38585 (local address 127.0.0.1:60540)
0929 00:58:57.786411 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 00:58:57.786491 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 00:58:57.802551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18356
I20260929 00:58:57.819084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18005
I20260929 00:58:57.820144 18066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:58:57.929261 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18005
2026-09-29T00:58:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2847 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:58:58 dist-test-slave-pzsx krb5kdc[18538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:58:58 dist-test-slave-pzsx krb5kdc[18538](info): set up 2 sockets
Sep 29 00:58:58 dist-test-slave-pzsx krb5kdc[18538](info): commencing operation
krb5kdc: starting...
W20260929 00:58:59.044663 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:58:59 dist-test-slave-pzsx krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:58:59.288287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32941
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:58:59.617388 18551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:58:59.617697 18551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:58:59.617889 18551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:58:59.625232 18551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:58:59.625336 18551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:58:59.625439 18551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:58:59.625538 18551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:58:59.625623 18551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:58:59.635499 18551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:58:59.636873 18551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:58:59.638207 18551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:58:59.650167 18556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:59.650214 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:58:59.650079 18559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:58:59.651201 18551 server_base.cc:1034] running on GCE node
I20260929 00:58:59.651950 18551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:58:59.653445 18551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:58:59.654656 18551 hybrid_clock.cc:648] HybridClock initialized: now 1790643539654615 us; error 28 us; skew 500 ppm
I20260929 00:58:59.655031 18551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:58:59 dist-test-slave-pzsx krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:58:59.658231 18551 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:58:59.659845 18551 webserver.cc:460] Webserver started at http://127.0.97.62:34703/ using document root <none> and password file <none>
I20260929 00:58:59.660614 18551 fs_manager.cc:359] Metadata directory not provided
I20260929 00:58:59.660759 18551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:58:59.661060 18551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:58:59.664001 18551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "6397c9e5b0ad4434a3b9f4be51c1defb"
format_stamp: "Formatted at 2026-09-29 00:58:59 on dist-test-slave-pzsx"
server_key: "8770afb2aa642106972e43f821d0720c"
server_key_iv: "9d7b8584670b3343fa99de4db964837b"
server_key_version: "encryptionkey@0"
I20260929 00:58:59.664805 18551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "6397c9e5b0ad4434a3b9f4be51c1defb"
format_stamp: "Formatted at 2026-09-29 00:58:59 on dist-test-slave-pzsx"
server_key: "8770afb2aa642106972e43f821d0720c"
server_key_iv: "9d7b8584670b3343fa99de4db964837b"
server_key_version: "encryptionkey@0"
I20260929 00:58:59.670027 18551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 00:58:59.673642 18566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:58:59.674970 18551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:58:59.675161 18551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "6397c9e5b0ad4434a3b9f4be51c1defb"
format_stamp: "Formatted at 2026-09-29 00:58:59 on dist-test-slave-pzsx"
server_key: "8770afb2aa642106972e43f821d0720c"
server_key_iv: "9d7b8584670b3343fa99de4db964837b"
server_key_version: "encryptionkey@0"
I20260929 00:58:59.675369 18551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:58:59.699316 18551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:58:59.703547 18551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:58:59.703840 18551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:58:59.722610 18551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32941
I20260929 00:58:59.722697 18617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32941 every 8 connection(s)
I20260929 00:58:59.724041 18551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:58:59.727686 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:58:59.732208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18551
I20260929 00:58:59.732407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:58:59.732859 388 external_mini_cluster.cc:1428] Setting key ad5a8598804e0b2cbd0469d20bfa5826
I20260929 00:58:59.738442 18618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: Bootstrap starting.
I20260929 00:58:59.742141 18618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: Neither blocks nor log segments found. Creating new log.
I20260929 00:58:59.743362 18618 log.cc:824] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: Log is configured to *not* fsync() on all Append() calls
I20260929 00:58:59.746882 18618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: No bootstrap required, opened a new log
Sep 29 00:58:59 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:58:59.757537 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:58:59.734708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.0.97.62:32941)
0929 00:58:59.735543 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 00:58:59.735555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:58:59.735594 (+ 39us) server_negotiation.cc:407] Connection header received
0929 00:58:59.737378 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:58:59.737400 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:58:59.737790 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:58:59.738100 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:58:59.739811 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:59.741210 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:59.743126 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:58:59.743907 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:58:59.744123 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:58:59.748868 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:58:59.748878 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:58:59.748899 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:58:59.748946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:58:59.752986 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:59.753675 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:59.753685 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:59.753726 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:59.753930 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:58:59.754323 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:58:59.754328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:58:59.754333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:58:59.755040 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:58:59.755352 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 00:58:59.755637 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 00:58:59.756112 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":389,"threads_started":1}
I20260929 00:58:59.757900 18618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } }
I20260929 00:58:59.758294 18618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:58:59.758410 18618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6397c9e5b0ad4434a3b9f4be51c1defb, State: Initialized, Role: FOLLOWER
I20260929 00:58:59.758955 18618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } }
I20260929 00:58:59.759222 18618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:58:59.759377 18618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:58:59.759568 18618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:58:59.761953 18618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } }
I20260929 00:58:59.762444 18618 leader_election.cc:302] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6397c9e5b0ad4434a3b9f4be51c1defb; no voters:
I20260929 00:58:59.763031 18618 leader_election.cc:288] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:58:59.763159 18623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:58:59.763885 18623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [term 1 LEADER]: Becoming Leader. State: Replica: 6397c9e5b0ad4434a3b9f4be51c1defb, State: Running, Role: LEADER
I20260929 00:58:59.764460 18623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } }
I20260929 00:58:59.765267 18618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:58:59.767537 18625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6397c9e5b0ad4434a3b9f4be51c1defb. Latest consensus state: current_term: 1 leader_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } } }
I20260929 00:58:59.767621 18624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6397c9e5b0ad4434a3b9f4be51c1defb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32941 } } }
I20260929 00:58:59.767976 18625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:59.767976 18624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb [sys.catalog]: This master's current role is: LEADER
I20260929 00:58:59.769339 18630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:58:59.774184 18630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:58:59.784433 18630 catalog_manager.cc:1347] Generated new cluster ID: 6b2ff1606af44d17b5eac9e2f32031a1
I20260929 00:58:59.784570 18630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:58:59.792263 18630 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:58:59.793596 18630 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:58:59.800904 18630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: Generated new TSK 0
I20260929 00:58:59.801697 18630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:58:59.858938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:00.191475 18644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:00.191740 18644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:00.191833 18644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:00.198850 18644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:00.198957 18644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:00.199069 18644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:59:00.208813 18644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:00.210026 18644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:00.211372 18644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:00.224018 18649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:00.224068 18650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:00.224107 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:00.225092 18644 server_base.cc:1034] running on GCE node
I20260929 00:59:00.225719 18644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:00.226580 18644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:00.227820 18644 hybrid_clock.cc:648] HybridClock initialized: now 1790643540227747 us; error 56 us; skew 500 ppm
I20260929 00:59:00.228161 18644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:00 dist-test-slave-pzsx krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:00.231720 18644 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:59:00.233223 18644 webserver.cc:460] Webserver started at http://127.0.97.1:42227/ using document root <none> and password file <none>
I20260929 00:59:00.233928 18644 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:00.234064 18644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:00.234388 18644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:00.237406 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "f86bdff4c6ce4664af74ebbb4d7235bc"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "780ef5ee4857cb12608509cbcdb90136"
server_key_iv: "2b8e565da450614792193948d676a9f9"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.238237 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f86bdff4c6ce4664af74ebbb4d7235bc"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "780ef5ee4857cb12608509cbcdb90136"
server_key_iv: "2b8e565da450614792193948d676a9f9"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.243567 18644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 00:59:00.247128 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.248574 18644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:00.248733 18644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "f86bdff4c6ce4664af74ebbb4d7235bc"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "780ef5ee4857cb12608509cbcdb90136"
server_key_iv: "2b8e565da450614792193948d676a9f9"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.248934 18644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:00.290392 18644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:00.294502 18644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:00.294808 18644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:00.295773 18644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:00.297171 18644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:00.297245 18644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.297389 18644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:00.297456 18644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.329842 18644 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44819
I20260929 00:59:00.329919 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44819 every 8 connection(s)
I20260929 00:59:00.331168 18644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:00.332991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18644
I20260929 00:59:00.333177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:00.333673 388 external_mini_cluster.cc:1428] Setting key 5224dfc4627de1384aaf23e1e7932b1c
Sep 29 00:59:00 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:00.356240 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:00.334003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35237 (local address 127.0.97.62:32941)
0929 00:59:00.334749 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:00.334753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:00.335636 (+ 883us) server_negotiation.cc:407] Connection header received
0929 00:59:00.338596 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:00.338601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:00.338662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:00.338756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:00.340856 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.341577 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.344075 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.344456 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.344654 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:00.348527 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:00.348536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:00.348544 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:00.348590 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:00.352089 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:00.352940 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:00.352948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:00.352955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:00.353103 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:00.353636 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:00.353640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:00.353642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:00.353923 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:00.354039 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:00.355127 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0929 00:59:00.355530 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":487,"threads_started":1}
I20260929 00:59:00.357491 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:00.334685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32941 (local address 127.0.97.1:35237)
0929 00:59:00.335480 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:00.335536 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:00.338283 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:00.339031 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:00.339046 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:00.339811 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:00.340672 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:00.340701 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.341747 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.341755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:00.343918 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:00.343937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.344762 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.344772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:00.345109 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:00.345896 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:00.345918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:00.348268 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:00.352262 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:00.352269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:00.352283 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:00.352821 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:00.353249 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:00.353256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:00.353263 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:00.353511 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:00.354070 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:00.354081 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:00.354547 (+ 466us) client_negotiation.cc:769] Sending connection context
0929 00:59:00.354878 (+ 331us) client_negotiation.cc:240] Negotiation successful
0929 00:59:00.355475 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
I20260929 00:59:00.359588 18772 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32941
I20260929 00:59:00.360232 18772 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:00.361251 18772 heartbeater.cc:511] Master 127.0.97.62:32941 requested a full tablet report, sending...
I20260929 00:59:00.364296 18583 ts_manager.cc:194] Registered new tserver with Master: f86bdff4c6ce4664af74ebbb4d7235bc (127.0.97.1:44819)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 00:59:00.366504 18583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:59:00.400259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:00.736028 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:00.736294 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:00.736400 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:00.743820 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:00.743927 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:00.744065 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:59:00.754297 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:00.755488 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:00.756760 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:00.769729 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:00.770108 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:00.770149 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:00.771274 18778 server_base.cc:1034] running on GCE node
I20260929 00:59:00.771862 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:00.772819 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:00.774039 18778 hybrid_clock.cc:648] HybridClock initialized: now 1790643540773978 us; error 72 us; skew 500 ppm
I20260929 00:59:00.774354 18778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:00 dist-test-slave-pzsx krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:00.780939 18778 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:59:00.782315 18778 webserver.cc:460] Webserver started at http://127.0.97.2:37059/ using document root <none> and password file <none>
I20260929 00:59:00.782944 18778 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:00.783044 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:00.783304 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:00.786212 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "e612487379064933921bffd08f082635"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "4b4a2ebc427996ef55fe943bfa1f8839"
server_key_iv: "08759cf2a34efc239768eecc63872e64"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.786909 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e612487379064933921bffd08f082635"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "4b4a2ebc427996ef55fe943bfa1f8839"
server_key_iv: "08759cf2a34efc239768eecc63872e64"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.791792 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:00.795030 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.796346 18778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:00.796487 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "e612487379064933921bffd08f082635"
format_stamp: "Formatted at 2026-09-29 00:59:00 on dist-test-slave-pzsx"
server_key: "4b4a2ebc427996ef55fe943bfa1f8839"
server_key_iv: "08759cf2a34efc239768eecc63872e64"
server_key_version: "encryptionkey@0"
I20260929 00:59:00.796790 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:00.809805 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:00.813557 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:00.813809 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:00.814661 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:00.815905 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:00.815975 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.816057 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:00.816100 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:00.849301 18778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45851
I20260929 00:59:00.849342 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45851 every 8 connection(s)
I20260929 00:59:00.850620 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:00.851955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18778
I20260929 00:59:00.852144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:00.852737 388 external_mini_cluster.cc:1428] Setting key 616004966853bcc57fd4be11d035a213
Sep 29 00:59:00 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:00.872159 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:00.853347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35933 (local address 127.0.97.62:32941)
0929 00:59:00.853560 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:00.853566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:00.855000 (+ 1434us) server_negotiation.cc:407] Connection header received
0929 00:59:00.856932 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:00.856939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:00.857058 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:00.857174 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:00.858743 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.859457 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.861097 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.861621 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.861781 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:00.865875 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:00.865880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:00.865884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:00.865917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:00.868621 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:00.869640 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:00.869644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:00.869646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:00.869833 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:00.870366 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:00.870369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:00.870371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:00.870649 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:00.870795 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:00.871735 (+ 940us) server_negotiation.cc:299] Negotiation successful
0929 00:59:00.871911 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:59:00.873854 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:00.853968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32941 (local address 127.0.97.2:35933)
0929 00:59:00.854818 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:00.854868 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:00.856666 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:00.857424 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:00.857432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:00.857970 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:00.858551 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:00.858568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.859611 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.859616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:00.860943 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:00.860956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:00.861874 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:00.861880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:00.862096 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:00.862807 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:00.862835 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:00.865580 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:00.868768 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:00.868780 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:00.868800 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:00.869514 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:00.869962 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:00.869969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:00.869976 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:00.870248 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:00.870812 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:00.870823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:00.871324 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 00:59:00.871650 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 00:59:00.872270 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":321,"threads_started":1}
I20260929 00:59:00.875463 18906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32941
I20260929 00:59:00.875772 18906 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:00.876580 18906 heartbeater.cc:511] Master 127.0.97.62:32941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 00:59:00.878084 18583 ts_manager.cc:194] Registered new tserver with Master: e612487379064933921bffd08f082635 (127.0.97.2:45851)
I20260929 00:59:00.878978 18583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:59:00.908058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:01.233562 18911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:01.233899 18911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:01.234066 18911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:01.241393 18911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:01.241554 18911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:01.241734 18911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:59:01.251485 18911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:01.252949 18911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:01.254278 18911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:01.267221 18916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:01.267256 18919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:01.267179 18917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:01.267967 18911 server_base.cc:1034] running on GCE node
I20260929 00:59:01.268627 18911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:01.269568 18911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:01.270826 18911 hybrid_clock.cc:648] HybridClock initialized: now 1790643541270771 us; error 40 us; skew 500 ppm
I20260929 00:59:01.271162 18911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:01 dist-test-slave-pzsx krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:01.274295 18911 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:59:01.275759 18911 webserver.cc:460] Webserver started at http://127.0.97.3:39757/ using document root <none> and password file <none>
I20260929 00:59:01.276397 18911 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:01.276544 18911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:01.276849 18911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:01.279917 18911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "25309ade65b24d5cb469b11499d8ac7c"
format_stamp: "Formatted at 2026-09-29 00:59:01 on dist-test-slave-pzsx"
server_key: "9320f368e1bcfe159591880fa0dd2ed2"
server_key_iv: "152afc1b857ff614b60a8093240d810f"
server_key_version: "encryptionkey@0"
I20260929 00:59:01.280730 18911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "25309ade65b24d5cb469b11499d8ac7c"
format_stamp: "Formatted at 2026-09-29 00:59:01 on dist-test-slave-pzsx"
server_key: "9320f368e1bcfe159591880fa0dd2ed2"
server_key_iv: "152afc1b857ff614b60a8093240d810f"
server_key_version: "encryptionkey@0"
I20260929 00:59:01.285634 18911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:59:01.288934 18926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:01.290194 18911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:59:01.290357 18911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "25309ade65b24d5cb469b11499d8ac7c"
format_stamp: "Formatted at 2026-09-29 00:59:01 on dist-test-slave-pzsx"
server_key: "9320f368e1bcfe159591880fa0dd2ed2"
server_key_iv: "152afc1b857ff614b60a8093240d810f"
server_key_version: "encryptionkey@0"
I20260929 00:59:01.290547 18911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:01.310403 18911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:01.314193 18911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:01.314479 18911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:01.315411 18911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:01.316838 18911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:01.316910 18911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:01.317085 18911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:01.317135 18911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:01.350236 18911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42145
I20260929 00:59:01.350310 19038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42145 every 8 connection(s)
I20260929 00:59:01.351523 18911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:01.352655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 18911
I20260929 00:59:01.352874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:01.353583 388 external_mini_cluster.cc:1428] Setting key b90ad942cb96d43fbfbba2258af704f8
Sep 29 00:59:01 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:01.370499 18772 heartbeater.cc:503] Master 127.0.97.62:32941 was elected leader, sending a full tablet report...
I20260929 00:59:01.371889 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.354501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45565 (local address 127.0.97.62:32941)
0929 00:59:01.354695 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.354698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.355916 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 00:59:01.357805 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.357810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.357876 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.357958 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:01.359382 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.360157 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.361673 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.362064 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.362156 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.365496 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:01.365503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:01.365507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:01.365551 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:01.368736 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.369503 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.369508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.369512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.369653 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.370150 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.370155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.370159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.370421 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:01.370576 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.371305 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.371528 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:59:01.372918 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.354950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32941 (local address 127.0.97.3:45565)
0929 00:59:01.355752 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.355804 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.357563 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.358160 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.358169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.358648 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:01.359235 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.359260 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.360278 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.360281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.361546 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.361558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.362244 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.362248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.362435 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.363043 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:01.363069 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:01.365285 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:01.368869 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:01.368880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:01.368895 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:01.369364 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:01.369780 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:01.369784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:01.369788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:01.370032 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:01.370555 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:01.370563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:01.370911 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.371169 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.371712 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":337,"threads_started":1}
I20260929 00:59:01.374492 19039 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32941
I20260929 00:59:01.374846 19039 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:01.375572 19039 heartbeater.cc:511] Master 127.0.97.62:32941 requested a full tablet report, sending...
I20260929 00:59:01.376952 18583 ts_manager.cc:194] Registered new tserver with Master: 25309ade65b24d5cb469b11499d8ac7c (127.0.97.3:42145)
I20260929 00:59:01.377683 18583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45565
I20260929 00:59:01.380965 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:01.397035 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.384724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.0.97.62:32941)
0929 00:59:01.384876 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.384881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.385390 (+ 509us) server_negotiation.cc:407] Connection header received
0929 00:59:01.385517 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.385520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.385576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.385652 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:01.386973 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.387658 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.388931 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.389324 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.389413 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.391781 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:01.391785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:01.391788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:01.391817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:01.394536 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.395319 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.395322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.395324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.395434 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.395822 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.395825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.395827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.396119 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:01.396316 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.396674 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.396818 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:01.400900 18583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:01.407518 18583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:01.432185 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.420719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.0.97.2:45851)
0929 00:59:01.422028 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.422034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.422942 (+ 908us) server_negotiation.cc:407] Connection header received
0929 00:59:01.423124 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.423131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.423354 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.423548 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:01.424479 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.426623 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.429168 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.430562 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.430651 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.430808 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.431669 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.431844 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":1059,"threads_started":1}
I20260929 00:59:01.432235 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.420323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57304)
0929 00:59:01.421097 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.421148 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.421324 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.423426 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.423433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.423469 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:01.423970 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.423982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.426131 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.426136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.428688 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.428703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.430955 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.430963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.431291 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.431459 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.431597 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.431787 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":177,"thread_start_us":558,"threads_started":1}
I20260929 00:59:01.433194 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.420545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45851 (local address 127.0.0.1:48950)
0929 00:59:01.422754 (+ 2209us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.422773 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.422956 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.423775 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.423780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.423812 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:01.424272 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.424286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.426834 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.426839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.428992 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.429013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.430985 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.430991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.431291 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.431459 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.432640 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.432776 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"mutex_wait_us":99,"thread_start_us":1865,"threads_started":1}
I20260929 00:59:01.433801 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.421270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.97.1:44819)
0929 00:59:01.422472 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.422482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.422522 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:59:01.422674 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.422683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.422956 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.423157 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:01.424214 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.425961 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.428877 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.430407 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.430547 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.430795 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.433275 (+ 2480us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.433448 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":896,"threads_started":1}
I20260929 00:59:01.438083 18707 tablet_service.cc:1467] Processing CreateTablet for tablet 123132c848564efc88cc3d30ad4faf23 (DEFAULT_TABLE table=test-table [id=d5566771b4f643b68518177900c62a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:01.438584 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 123132c848564efc88cc3d30ad4faf23 (DEFAULT_TABLE table=test-table [id=d5566771b4f643b68518177900c62a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:01.440044 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123132c848564efc88cc3d30ad4faf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:01.440323 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123132c848564efc88cc3d30ad4faf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:01.441386 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.423031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42145 (local address 127.0.0.1:56922)
0929 00:59:01.432478 (+ 9447us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.432531 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.432631 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.435055 (+ 2424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.435064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.435101 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:01.435355 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.435369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.437016 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.437022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.438752 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.438770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.440599 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.440607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.440785 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.440821 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.440929 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.441091 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9347,"thread_start_us":12394,"threads_started":1}
I20260929 00:59:01.441442 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.423476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.97.3:42145)
0929 00:59:01.434496 (+ 11020us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.434510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.434536 (+ 26us) server_negotiation.cc:407] Connection header received
0929 00:59:01.434610 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.434615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.434854 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.435054 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:01.435570 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.436843 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.438933 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.440426 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.440588 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.440813 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.440970 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.441125 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10879,"thread_start_us":10739,"threads_started":1}
I20260929 00:59:01.445300 18974 tablet_service.cc:1467] Processing CreateTablet for tablet 123132c848564efc88cc3d30ad4faf23 (DEFAULT_TABLE table=test-table [id=d5566771b4f643b68518177900c62a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:01.446750 18974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123132c848564efc88cc3d30ad4faf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:01.452793 19057 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Bootstrap starting.
I20260929 00:59:01.456830 19057 tablet_bootstrap.cc:654] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:01.457970 19057 log.cc:824] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:01.461478 19057 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: No bootstrap required, opened a new log
I20260929 00:59:01.462845 19059 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Bootstrap starting.
I20260929 00:59:01.463411 19057 ts_tablet_manager.cc:1397] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260929 00:59:01.467350 19059 tablet_bootstrap.cc:654] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:01.467829 19060 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Bootstrap starting.
I20260929 00:59:01.468622 19059 log.cc:824] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:01.471815 19060 tablet_bootstrap.cc:654] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:01.473269 19060 log.cc:824] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:01.474599 19057 raft_consensus.cc:348] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.475003 19057 raft_consensus.cc:374] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:01.475131 19057 raft_consensus.cc:729] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86bdff4c6ce4664af74ebbb4d7235bc, State: Initialized, Role: FOLLOWER
I20260929 00:59:01.475749 19057 consensus_queue.cc:260] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.476214 19059 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: No bootstrap required, opened a new log
I20260929 00:59:01.476572 19059 ts_tablet_manager.cc:1397] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260929 00:59:01.477564 19057 ts_tablet_manager.cc:1428] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260929 00:59:01.480476 19060 tablet_bootstrap.cc:492] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: No bootstrap required, opened a new log
I20260929 00:59:01.480773 19060 ts_tablet_manager.cc:1397] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260929 00:59:01.487149 19059 raft_consensus.cc:348] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.487637 19059 raft_consensus.cc:374] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:01.487746 19059 raft_consensus.cc:729] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612487379064933921bffd08f082635, State: Initialized, Role: FOLLOWER
I20260929 00:59:01.488337 19059 consensus_queue.cc:260] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.489374 18906 heartbeater.cc:503] Master 127.0.97.62:32941 was elected leader, sending a full tablet report...
I20260929 00:59:01.490067 19059 ts_tablet_manager.cc:1428] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 00:59:01.491034 19060 raft_consensus.cc:348] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.491351 19060 raft_consensus.cc:374] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:01.491464 19060 raft_consensus.cc:729] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25309ade65b24d5cb469b11499d8ac7c, State: Initialized, Role: FOLLOWER
I20260929 00:59:01.492008 19060 consensus_queue.cc:260] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.493103 19039 heartbeater.cc:503] Master 127.0.97.62:32941 was elected leader, sending a full tablet report...
I20260929 00:59:01.493662 19060 ts_tablet_manager.cc:1428] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260929 00:59:01.495033 19063 raft_consensus.cc:482] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:01.495324 19063 raft_consensus.cc:504] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.497234 19063 leader_election.cc:288] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e612487379064933921bffd08f082635 (127.0.97.2:45851), 25309ade65b24d5cb469b11499d8ac7c (127.0.97.3:42145)
I20260929 00:59:01.506102 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.497571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46413 (local address 127.0.97.2:45851)
0929 00:59:01.497737 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.497741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.498396 (+ 655us) server_negotiation.cc:407] Connection header received
0929 00:59:01.498697 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.498704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.498784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.498848 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:01.499652 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.500814 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.503622 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.504877 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.504995 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.505087 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.505728 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.505853 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:59:01.506102 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.498575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36505 (local address 127.0.97.3:42145)
0929 00:59:01.498756 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.498761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.498998 (+ 237us) server_negotiation.cc:407] Connection header received
0929 00:59:01.499141 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.499144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.499202 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.499271 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:01.499882 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.500952 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.503622 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.504873 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.504995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.505090 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.505728 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.505853 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:01.506330 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.498326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42145 (local address 127.0.97.1:36505)
0929 00:59:01.498910 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.498926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.499041 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.499326 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.499334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.499363 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:01.499755 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.499764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.501077 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.501083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.503462 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.503478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.505126 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.505150 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.505466 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.505593 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.505688 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.505865 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":215,"thread_start_us":289,"threads_started":1}
I20260929 00:59:01.506418 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.497598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45851 (local address 127.0.97.1:46413)
0929 00:59:01.498287 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.498305 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:01.498489 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:01.499061 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:01.499067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:01.499095 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:01.499513 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.499528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.500919 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.500923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.503463 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:01.503478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.505161 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.505165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:01.505465 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.505590 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 00:59:01.505688 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 00:59:01.505865 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":217,"thread_start_us":339,"threads_started":1}
I20260929 00:59:01.507409 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123132c848564efc88cc3d30ad4faf23" candidate_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612487379064933921bffd08f082635" is_pre_election: true
I20260929 00:59:01.507457 18994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123132c848564efc88cc3d30ad4faf23" candidate_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25309ade65b24d5cb469b11499d8ac7c" is_pre_election: true
I20260929 00:59:01.507845 18861 raft_consensus.cc:2454] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86bdff4c6ce4664af74ebbb4d7235bc in term 0.
I20260929 00:59:01.507892 18994 raft_consensus.cc:2454] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86bdff4c6ce4664af74ebbb4d7235bc in term 0.
I20260929 00:59:01.508571 18662 leader_election.cc:302] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e612487379064933921bffd08f082635, f86bdff4c6ce4664af74ebbb4d7235bc; no voters:
I20260929 00:59:01.508934 19063 raft_consensus.cc:2781] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:01.509080 19063 raft_consensus.cc:482] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:01.509205 19063 raft_consensus.cc:3037] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:01.511364 19063 raft_consensus.cc:504] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.511929 19063 leader_election.cc:288] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [CANDIDATE]: Term 1 election: Requested vote from peers e612487379064933921bffd08f082635 (127.0.97.2:45851), 25309ade65b24d5cb469b11499d8ac7c (127.0.97.3:42145)
I20260929 00:59:01.512333 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123132c848564efc88cc3d30ad4faf23" candidate_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612487379064933921bffd08f082635"
I20260929 00:59:01.512427 18994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123132c848564efc88cc3d30ad4faf23" candidate_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25309ade65b24d5cb469b11499d8ac7c"
I20260929 00:59:01.512562 18861 raft_consensus.cc:3037] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:01.512656 18994 raft_consensus.cc:3037] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:01.515017 18994 raft_consensus.cc:2454] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86bdff4c6ce4664af74ebbb4d7235bc in term 1.
I20260929 00:59:01.515151 18861 raft_consensus.cc:2454] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86bdff4c6ce4664af74ebbb4d7235bc in term 1.
I20260929 00:59:01.515451 18662 leader_election.cc:302] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25309ade65b24d5cb469b11499d8ac7c, f86bdff4c6ce4664af74ebbb4d7235bc; no voters:
I20260929 00:59:01.515749 19063 raft_consensus.cc:2781] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:01.516189 19063 raft_consensus.cc:686] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 1 LEADER]: Becoming Leader. State: Replica: f86bdff4c6ce4664af74ebbb4d7235bc, State: Running, Role: LEADER
I20260929 00:59:01.516782 19063 consensus_queue.cc:237] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } }
I20260929 00:59:01.521551 18583 catalog_manager.cc:5495] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f86bdff4c6ce4664af74ebbb4d7235bc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86bdff4c6ce4664af74ebbb4d7235bc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:01.557751 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.552648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.0.97.1:44819)
0929 00:59:01.552860 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.552864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.552882 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:59:01.552965 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.552969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.553076 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.553221 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:01.553617 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.554330 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.555638 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.556028 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.556115 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.556639 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:01.557132 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:01.557266 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.557392 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.557460 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 00:59:01.565783 18861 raft_consensus.cc:1261] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 1 FOLLOWER]: Refusing update from remote peer f86bdff4c6ce4664af74ebbb4d7235bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:01.565871 18994 raft_consensus.cc:1261] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 1 FOLLOWER]: Refusing update from remote peer f86bdff4c6ce4664af74ebbb4d7235bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:01.566735 19068 consensus_queue.cc:1035] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e612487379064933921bffd08f082635" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:01.567137 19063 consensus_queue.cc:1035] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "25309ade65b24d5cb469b11499d8ac7c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:01.577243 19071 mvcc.cc:205] Tried to move back new op lower bound from 7334475946243420160 to 7334475946056327168. Current Snapshot: MvccSnapshot[applied={T|T < 7334475946243420160}]
I20260929 00:59:01.592581 18582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:01.592895 18582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:01.596554 18582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6397c9e5b0ad4434a3b9f4be51c1defb: Sending DeleteTablet for 3 replicas of tablet 123132c848564efc88cc3d30ad4faf23
I20260929 00:59:01.597728 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet 123132c848564efc88cc3d30ad4faf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57304
I20260929 00:59:01.598016 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 123132c848564efc88cc3d30ad4faf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48950
I20260929 00:59:01.598887 18974 tablet_service.cc:1514] Processing DeleteTablet for tablet 123132c848564efc88cc3d30ad4faf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56922
I20260929 00:59:01.601783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18644
I20260929 00:59:01.623950 19083 tablet_replica.cc:330] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: stopping tablet replica
I20260929 00:59:01.624430 19083 raft_consensus.cc:2229] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:01.625087 19083 raft_consensus.cc:2258] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:01.627480 19083 ts_tablet_manager.cc:1905] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:01.629034 19082 tablet_replica.cc:330] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: stopping tablet replica
I20260929 00:59:01.629460 19082 raft_consensus.cc:2229] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:01.629944 19082 raft_consensus.cc:2258] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:01.631917 19082 ts_tablet_manager.cc:1905] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:01.634804 19083 ts_tablet_manager.cc:1918] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:01.634984 19083 log.cc:1196] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/123132c848564efc88cc3d30ad4faf23
I20260929 00:59:01.635548 19083 ts_tablet_manager.cc:1939] T 123132c848564efc88cc3d30ad4faf23 P 25309ade65b24d5cb469b11499d8ac7c: Deleting consensus metadata
Sep 29 00:59:01 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:59:01.637488 18569 catalog_manager.cc:4841] TS 25309ade65b24d5cb469b11499d8ac7c (127.0.97.3:42145): tablet 123132c848564efc88cc3d30ad4faf23 (table test-table [id=d5566771b4f643b68518177900c62a94]) successfully deleted
I20260929 00:59:01.637997 19081 tablet_replica.cc:330] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: stopping tablet replica
I20260929 00:59:01.638482 19081 raft_consensus.cc:2229] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:01.638964 19082 ts_tablet_manager.cc:1918] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:01.639119 19082 log.cc:1196] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/123132c848564efc88cc3d30ad4faf23
I20260929 00:59:01.639210 19081 raft_consensus.cc:2258] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:01.639664 19082 ts_tablet_manager.cc:1939] T 123132c848564efc88cc3d30ad4faf23 P e612487379064933921bffd08f082635: Deleting consensus metadata
I20260929 00:59:01.641433 19081 ts_tablet_manager.cc:1905] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:01.643102 18569 catalog_manager.cc:4841] TS e612487379064933921bffd08f082635 (127.0.97.2:45851): tablet 123132c848564efc88cc3d30ad4faf23 (table test-table [id=d5566771b4f643b68518177900c62a94]) successfully deleted
I20260929 00:59:01.646310 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.603205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.0.97.1:44819)
0929 00:59:01.603381 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.603386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.619985 (+ 16599us) server_negotiation.cc:407] Connection header received
0929 00:59:01.620155 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.620164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.620303 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.620439 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:01.622730 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.624549 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.627484 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.628484 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.628623 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.638155 (+ 9532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:01.638168 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:01.638185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:01.638232 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:01.642698 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.643685 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.643694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.643698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.643876 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.644331 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.644336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.644340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.644834 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:01.645140 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.645624 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.645924 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:59:01.647099 18767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:01.746824 19081 ts_tablet_manager.cc:1918] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:01.747107 19081 log.cc:1196] T 123132c848564efc88cc3d30ad4faf23 P f86bdff4c6ce4664af74ebbb4d7235bc: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/123132c848564efc88cc3d30ad4faf23
I20260929 00:59:01.747427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18644
W20260929 00:59:01.765712 18570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44819 (error 108)
I20260929 00:59:01.765928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18778
I20260929 00:59:01.767508 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.767127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57326)
0929 00:59:01.767275 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.767364 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 00:59:01.767726 18570 catalog_manager.cc:4584] TS f86bdff4c6ce4664af74ebbb4d7235bc (127.0.97.1:44819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 123132c848564efc88cc3d30ad4faf23: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Sep 29 00:59:01 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:59:01.781551 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.766854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.0.97.2:45851)
0929 00:59:01.767037 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.767041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.767061 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:59:01.767724 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.767731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.767819 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.767934 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:01.769268 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.770133 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.771401 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.771783 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.771914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.776009 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:01.776022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:01.776037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:01.776069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:01.779064 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.779683 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.779689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.779691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.779802 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.780183 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.780186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.780188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.780596 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:01.780803 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.781059 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.781212 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 00:59:01.782274 18901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:01.821642 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.821157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57332)
0929 00:59:01.821398 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.821500 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 00:59:01.822688 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.822334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57342)
0929 00:59:01.822481 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.822553 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 00:59:01.871981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18778
I20260929 00:59:01.883153 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.882733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57344)
0929 00:59:01.882931 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.883019 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 00:59:01.884164 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.883805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57358)
0929 00:59:01.883948 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.884030 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 00:59:01.889078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18911
Sep 29 00:59:01 dist-test-slave-pzsx krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:59:01.903965 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.889761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.97.3:42145)
0929 00:59:01.889924 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:01.889928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:01.889946 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:59:01.890004 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:01.890008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:01.890071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:01.890154 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:01.891420 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.892110 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.893464 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:01.893848 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:01.893986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:01.898499 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:01.898532 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:01.898549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:01.898588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:01.901528 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.902159 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.902166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.902168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.902292 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:01.902600 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:01.902603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:01.902605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:01.902939 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:01.903148 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:01.903406 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 00:59:01.903587 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 00:59:01.904693 19034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:01.964753 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.964123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57374)
0929 00:59:01.964437 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.964606 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 00:59:01.965849 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:01.965453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44819 (local address 127.0.0.1:57382)
0929 00:59:01.965672 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:01.965746 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 00:59:01.993075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18911
I20260929 00:59:02.009685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 18551
I20260929 00:59:02.010778 18613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:02.123253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 18551
2026-09-29T00:59:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T00:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:02Z Disabled control of system clock
I20260929 00:59:02.198652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43287
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:59:02.520998 19093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:02.521277 19093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:02.521359 19093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:02.528555 19093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:02.528663 19093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:02.528699 19093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:02.528755 19093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:02.528798 19093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:02.538009 19093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:02.539378 19093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:02.541174 19093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:02.552848 19099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:02.552799 19101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:02.553007 19098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:02.553637 19093 server_base.cc:1034] running on GCE node
I20260929 00:59:02.554435 19093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:02.555745 19093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:02.556975 19093 hybrid_clock.cc:648] HybridClock initialized: now 1790643542556938 us; error 53 us; skew 500 ppm
I20260929 00:59:02.557382 19093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:02.559657 19093 webserver.cc:460] Webserver started at http://127.0.97.62:36975/ using document root <none> and password file <none>
I20260929 00:59:02.560325 19093 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:02.560451 19093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:02.560793 19093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:02.563551 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "9754770d1e19403c9f9f4da8d1db3b4e"
format_stamp: "Formatted at 2026-09-29 00:59:02 on dist-test-slave-pzsx"
server_key: "72d93d32ae9dc7e9530386d82fab71e7"
server_key_iv: "7611a2605798cba355d03c6fcdf3ffe1"
server_key_version: "encryptionkey@0"
I20260929 00:59:02.564317 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "9754770d1e19403c9f9f4da8d1db3b4e"
format_stamp: "Formatted at 2026-09-29 00:59:02 on dist-test-slave-pzsx"
server_key: "72d93d32ae9dc7e9530386d82fab71e7"
server_key_iv: "7611a2605798cba355d03c6fcdf3ffe1"
server_key_version: "encryptionkey@0"
I20260929 00:59:02.569399 19093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 00:59:02.572619 19107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:02.573869 19093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:59:02.574051 19093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "9754770d1e19403c9f9f4da8d1db3b4e"
format_stamp: "Formatted at 2026-09-29 00:59:02 on dist-test-slave-pzsx"
server_key: "72d93d32ae9dc7e9530386d82fab71e7"
server_key_iv: "7611a2605798cba355d03c6fcdf3ffe1"
server_key_version: "encryptionkey@0"
I20260929 00:59:02.574259 19093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:02.598780 19093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:02.599663 19093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:02.599969 19093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:02.618232 19093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43287
I20260929 00:59:02.618252 19158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43287 every 8 connection(s)
I20260929 00:59:02.619570 19093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:02.620684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19093
I20260929 00:59:02.620848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:02.621277 388 external_mini_cluster.cc:1428] Setting key 58f3171884b7edc37929acf205815bcd
I20260929 00:59:02.623863 19159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:02.634147 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:02.623053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.97.62:43287)
0929 00:59:02.623998 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:02.624013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:02.624076 (+ 63us) server_negotiation.cc:407] Connection header received
0929 00:59:02.625857 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:02.625906 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:02.626346 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:02.626747 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:02.627307 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:02.628646 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:02.629973 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:02.630608 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:02.630827 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:02.631990 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:02.631998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:02.632014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:02.632605 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:02.632681 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:02.632701 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:02.632826 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:02.633008 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 00:59:02.633295 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
I20260929 00:59:02.634229 19159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: Bootstrap starting.
I20260929 00:59:02.637389 19159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:02.638347 19159 log.cc:824] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:02.640924 19159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: No bootstrap required, opened a new log
I20260929 00:59:02.647851 19159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } }
I20260929 00:59:02.648222 19159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:02.648326 19159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9754770d1e19403c9f9f4da8d1db3b4e, State: Initialized, Role: FOLLOWER
I20260929 00:59:02.648901 19159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } }
I20260929 00:59:02.649137 19159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:02.649278 19159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:02.649438 19159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:02.651584 19159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } }
I20260929 00:59:02.652047 19159 leader_election.cc:302] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9754770d1e19403c9f9f4da8d1db3b4e; no voters:
I20260929 00:59:02.652691 19159 leader_election.cc:288] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:02.652796 19164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:02.653517 19164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [term 1 LEADER]: Becoming Leader. State: Replica: 9754770d1e19403c9f9f4da8d1db3b4e, State: Running, Role: LEADER
I20260929 00:59:02.654047 19164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } }
I20260929 00:59:02.654635 19159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:02.656816 19166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9754770d1e19403c9f9f4da8d1db3b4e. Latest consensus state: current_term: 1 leader_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } } }
I20260929 00:59:02.656793 19165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754770d1e19403c9f9f4da8d1db3b4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43287 } } }
I20260929 00:59:02.657343 19166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:02.657406 19165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:02.658238 19169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:02.663817 19169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:02.672719 19169 catalog_manager.cc:1347] Generated new cluster ID: 536d8d55619a4421bf1020696f61f8cd
I20260929 00:59:02.672834 19169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:02.687469 19169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:02.688851 19169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:02.698454 19169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: Generated new TSK 0
I20260929 00:59:02.699256 19169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:02.705716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--builtin_ntp_servers=127.0.97.20:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:59:03.028885 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:03.029151 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:03.029325 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:03.036407 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:03.036520 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:03.036718 19183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:59:03.045607 19183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:03.046865 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:03.048619 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:03.061412 19188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.061764 19191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.061441 19189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:03.062577 19183 server_base.cc:1034] running on GCE node
I20260929 00:59:03.063086 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:03.063930 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:03.065152 19183 hybrid_clock.cc:648] HybridClock initialized: now 1790643543065103 us; error 44 us; skew 500 ppm
I20260929 00:59:03.065459 19183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:03.067728 19183 webserver.cc:460] Webserver started at http://127.0.97.1:35765/ using document root <none> and password file <none>
I20260929 00:59:03.068427 19183 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:03.068575 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:03.068897 19183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:03.071780 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "7bca7c229a9344e1a4dce70477cc5ccd"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "64c39d2aec1bcfb7d2b1bf69bfae13b2"
server_key_iv: "7d46bd6fe78369b9bd6dd69fdcd87323"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.072615 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7bca7c229a9344e1a4dce70477cc5ccd"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "64c39d2aec1bcfb7d2b1bf69bfae13b2"
server_key_iv: "7d46bd6fe78369b9bd6dd69fdcd87323"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.077598 19183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:59:03.081076 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.082247 19183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:03.082424 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "7bca7c229a9344e1a4dce70477cc5ccd"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "64c39d2aec1bcfb7d2b1bf69bfae13b2"
server_key_iv: "7d46bd6fe78369b9bd6dd69fdcd87323"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.082633 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:03.099161 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:03.099939 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:03.100221 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:03.101229 19183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:03.102632 19183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:03.102768 19183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.102886 19183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:03.102969 19183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.136224 19183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46341
I20260929 00:59:03.136276 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46341 every 8 connection(s)
I20260929 00:59:03.137629 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:03.141683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19183
I20260929 00:59:03.141853 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:03.142457 388 external_mini_cluster.cc:1428] Setting key 4ee9b700c631e59df89b954395843998
I20260929 00:59:03.145252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--builtin_ntp_servers=127.0.97.20:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:59:03.151845 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:03.140197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52153 (local address 127.0.97.62:43287)
0929 00:59:03.140705 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:03.140710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:03.142202 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 00:59:03.144210 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:03.144215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:03.144337 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:03.144459 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:03.145646 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.146857 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.148671 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.149380 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.149547 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:03.150870 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:03.150879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:03.150887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:03.150913 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:03.150966 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:03.150984 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:03.151074 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:03.151282 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 00:59:03.151398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":244,"threads_started":1}
I20260929 00:59:03.152623 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:03.140884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43287 (local address 127.0.97.1:52153)
0929 00:59:03.142002 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:03.142067 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:03.143927 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:03.144722 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:03.144731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:03.144822 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:03.145469 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:03.145496 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.147002 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.147011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:03.148432 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:03.148443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.149593 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.149598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:03.149793 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:03.150531 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:03.150534 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:03.150571 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:03.150647 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:03.151077 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:03.151085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:03.151091 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 00:59:03.151222 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 00:59:03.151515 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":286,"threads_started":1}
I20260929 00:59:03.154338 19310 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43287
I20260929 00:59:03.154664 19310 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:03.155591 19310 heartbeater.cc:511] Master 127.0.97.62:43287 requested a full tablet report, sending...
I20260929 00:59:03.157980 19124 ts_manager.cc:194] Registered new tserver with Master: 7bca7c229a9344e1a4dce70477cc5ccd (127.0.97.1:46341)
I20260929 00:59:03.159467 19124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52153
W20260929 00:59:03.467612 19314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:03.467859 19314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:03.467957 19314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:03.474851 19314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:03.474933 19314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:03.475059 19314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:59:03.483680 19314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:03.484958 19314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:03.486610 19314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:03.499153 19320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.499073 19319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.499112 19322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:03.499933 19314 server_base.cc:1034] running on GCE node
I20260929 00:59:03.500603 19314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:03.501487 19314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:03.502718 19314 hybrid_clock.cc:648] HybridClock initialized: now 1790643543502671 us; error 58 us; skew 500 ppm
I20260929 00:59:03.503026 19314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:03.505100 19314 webserver.cc:460] Webserver started at http://127.0.97.2:43749/ using document root <none> and password file <none>
I20260929 00:59:03.505725 19314 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:03.505844 19314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:03.506141 19314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:03.508903 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "8e862dbf218b4a879d19f607a8220a31"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "9108891a868f8f88d88ebc802c1cd6c7"
server_key_iv: "780c1c71207c38c66cc1b0fb2927be25"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.509699 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8e862dbf218b4a879d19f607a8220a31"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "9108891a868f8f88d88ebc802c1cd6c7"
server_key_iv: "780c1c71207c38c66cc1b0fb2927be25"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.514818 19314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 00:59:03.518194 19328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.519452 19314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:59:03.519641 19314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "8e862dbf218b4a879d19f607a8220a31"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "9108891a868f8f88d88ebc802c1cd6c7"
server_key_iv: "780c1c71207c38c66cc1b0fb2927be25"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.519860 19314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:03.534230 19314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:03.535001 19314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:03.535274 19314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:03.536195 19314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:03.537647 19314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:03.537765 19314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.537902 19314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:03.537990 19314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.570595 19314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35037
I20260929 00:59:03.570643 19440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35037 every 8 connection(s)
I20260929 00:59:03.571916 19314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:03.572379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19314
I20260929 00:59:03.572589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:03.573197 388 external_mini_cluster.cc:1428] Setting key bb22a330aca5a5a2f2a496aa0636fced
I20260929 00:59:03.576252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--builtin_ntp_servers=127.0.97.20:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:59:03.587424 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:03.574702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45923 (local address 127.0.97.62:43287)
0929 00:59:03.574894 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:03.574901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:03.576589 (+ 1688us) server_negotiation.cc:407] Connection header received
0929 00:59:03.579449 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:03.579454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:03.579529 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:03.579615 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:03.581072 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.581796 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.584215 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.584759 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.584883 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:03.586424 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:03.586434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:03.586442 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:03.586477 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:03.586519 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:03.586537 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:03.586633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:03.586884 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 00:59:03.586994 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:59:03.588274 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:03.575319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43287 (local address 127.0.97.2:45923)
0929 00:59:03.576415 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:03.576470 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:03.579159 (+ 2689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:03.579891 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:03.579904 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:03.580024 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:03.580883 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:03.580908 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.581956 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.581965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:03.584032 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:03.584049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:03.585003 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:03.585012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:03.585254 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:03.586103 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:03.586105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:03.586145 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:03.586226 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:03.586646 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:03.586658 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:03.586668 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 00:59:03.586870 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 00:59:03.587163 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260929 00:59:03.589815 19441 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43287
I20260929 00:59:03.590109 19441 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:03.590849 19441 heartbeater.cc:511] Master 127.0.97.62:43287 requested a full tablet report, sending...
I20260929 00:59:03.592257 19124 ts_manager.cc:194] Registered new tserver with Master: 8e862dbf218b4a879d19f607a8220a31 (127.0.97.2:35037)
I20260929 00:59:03.593549 19124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45923
W20260929 00:59:03.907045 19444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:03.907306 19444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:03.907388 19444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:03.914472 19444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:03.914574 19444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:03.914736 19444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:59:03.923774 19444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:03.925032 19444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:03.926744 19444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:03.939880 19452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.939953 19449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:03.939986 19450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:03.940815 19444 server_base.cc:1034] running on GCE node
I20260929 00:59:03.941421 19444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:03.942351 19444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:03.943584 19444 hybrid_clock.cc:648] HybridClock initialized: now 1790643543943544 us; error 53 us; skew 500 ppm
I20260929 00:59:03.943907 19444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:03.946141 19444 webserver.cc:460] Webserver started at http://127.0.97.3:45519/ using document root <none> and password file <none>
I20260929 00:59:03.946826 19444 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:03.947017 19444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:03.947362 19444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:03.950258 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ff0dfbd1e6746a1b2f444781d750d5e"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "a0e16cc4d948d40aa856165c6763a809"
server_key_iv: "0bfa9f7485f7b375fd5869e9c40240ef"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.951038 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ff0dfbd1e6746a1b2f444781d750d5e"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "a0e16cc4d948d40aa856165c6763a809"
server_key_iv: "0bfa9f7485f7b375fd5869e9c40240ef"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.955962 19444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:03.959332 19458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.960952 19444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:59:03.961161 19444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "9ff0dfbd1e6746a1b2f444781d750d5e"
format_stamp: "Formatted at 2026-09-29 00:59:03 on dist-test-slave-pzsx"
server_key: "a0e16cc4d948d40aa856165c6763a809"
server_key_iv: "0bfa9f7485f7b375fd5869e9c40240ef"
server_key_version: "encryptionkey@0"
I20260929 00:59:03.961400 19444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:03.981474 19444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:03.982312 19444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:03.982595 19444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:03.983544 19444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:03.985082 19444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:03.985212 19444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:03.985335 19444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:03.985430 19444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:04.018471 19444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36353
I20260929 00:59:04.018522 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36353 every 8 connection(s)
I20260929 00:59:04.019780 19444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:04.023630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19444
I20260929 00:59:04.023882 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:04.024412 388 external_mini_cluster.cc:1428] Setting key 8acb46eef362fe20827c3c764d498223
I20260929 00:59:04.033774 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.022732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43813 (local address 127.0.97.62:43287)
0929 00:59:04.022915 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.022919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.024736 (+ 1817us) server_negotiation.cc:407] Connection header received
0929 00:59:04.026850 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.026856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.026947 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.027062 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:04.028230 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.029012 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.030544 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.030964 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.031102 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.032884 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:04.032889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:04.032892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:04.032910 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:04.032943 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:04.032954 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:04.033058 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.033342 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.033437 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 00:59:04.035205 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.023360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43287 (local address 127.0.97.3:43813)
0929 00:59:04.024548 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.024607 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.026597 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.027302 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.027310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.027393 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:04.028058 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.028075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.029164 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.029169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.030385 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.030397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.031190 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.031198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.031526 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.032478 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:04.032482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:04.032552 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:04.032666 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:04.033094 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:04.033106 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:04.033114 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.033330 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.033709 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":414,"threads_started":1}
I20260929 00:59:04.036798 19571 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43287
I20260929 00:59:04.037132 19571 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:04.038143 19571 heartbeater.cc:511] Master 127.0.97.62:43287 requested a full tablet report, sending...
I20260929 00:59:04.039925 19124 ts_manager.cc:194] Registered new tserver with Master: 9ff0dfbd1e6746a1b2f444781d750d5e (127.0.97.3:36353)
I20260929 00:59:04.041112 19124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43813
I20260929 00:59:04.051921 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:04.062390 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.056127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.97.62:43287)
0929 00:59:04.056328 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.056337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.056748 (+ 411us) server_negotiation.cc:407] Connection header received
0929 00:59:04.056938 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.056946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.057035 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.057141 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:04.057630 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.058321 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.059800 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.060263 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.060396 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.061646 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:04.061653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:04.061658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:04.061685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:04.061717 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:04.061729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:04.061882 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.062064 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.062129 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 00:59:04.066438 19124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:04.073431 19124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:04.110926 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.085101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:40408)
0929 00:59:04.101324 (+ 16223us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.101381 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.101658 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.103618 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.103625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.103656 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:04.104162 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.104173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106124 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.106130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.108129 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.108140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.109979 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.109986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.110174 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.110220 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.110349 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.110550 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9185,"mutex_wait_us":161,"thread_start_us":16513,"threads_started":1}
I20260929 00:59:04.110916 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.085354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.97.1:46341)
0929 00:59:04.102982 (+ 17628us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.102991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.103019 (+ 28us) server_negotiation.cc:407] Connection header received
0929 00:59:04.103123 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.103132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.103403 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.103597 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:04.104363 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106016 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.108320 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.109797 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.109949 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.110220 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.110398 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.110561 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17448,"thread_start_us":17448,"threads_started":1}
I20260929 00:59:04.112392 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.095257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35037 (local address 127.0.0.1:39208)
0929 00:59:04.103052 (+ 7795us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.103069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.103181 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.103972 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.103976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.104020 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:04.104620 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.104636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106489 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.106494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.109205 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.109225 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.111647 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.111651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.111764 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.111785 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.111885 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.112018 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7675,"mutex_wait_us":230,"thread_start_us":7753,"threads_started":1}
I20260929 00:59:04.112454 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.095489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.0.97.2:35037)
0929 00:59:04.102658 (+ 7169us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.102669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.103183 (+ 514us) server_negotiation.cc:407] Connection header received
0929 00:59:04.103313 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.103319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.103536 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.103729 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:04.104853 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106300 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.109467 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.110889 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.111377 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.111547 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.111950 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.112103 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6807,"thread_start_us":6884,"threads_started":1}
I20260929 00:59:04.113056 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.085620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.97.3:36353)
0929 00:59:04.094139 (+ 8519us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.094150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.102158 (+ 8008us) server_negotiation.cc:407] Connection header received
0929 00:59:04.103292 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.103300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.103563 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.103713 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:04.104708 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106244 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.109468 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.110889 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.111019 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.111291 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.112460 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.112663 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8330,"thread_start_us":8170,"threads_started":1}
I20260929 00:59:04.113734 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.085227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36353 (local address 127.0.0.1:52864)
0929 00:59:04.102001 (+ 16774us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.102024 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.102914 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.103929 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.103939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.103982 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:04.104442 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.104450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.106384 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.106392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.109231 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.109240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.112067 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.112076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.112240 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.112281 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.113296 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.113422 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16586,"mutex_wait_us":195,"thread_start_us":8906,"threads_started":1}
I20260929 00:59:04.115061 19245 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e (DEFAULT_TABLE table=test-table [id=848761be5391408f8866b057a86530b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:04.115965 19376 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e (DEFAULT_TABLE table=test-table [id=848761be5391408f8866b057a86530b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:04.116660 19245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ce5e7ad1c476db3558a390a6a747e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:04.117131 19376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ce5e7ad1c476db3558a390a6a747e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:04.117390 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e (DEFAULT_TABLE table=test-table [id=848761be5391408f8866b057a86530b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:04.118697 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8ce5e7ad1c476db3558a390a6a747e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:04.128710 19589 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Bootstrap starting.
I20260929 00:59:04.131052 19590 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Bootstrap starting.
I20260929 00:59:04.132380 19591 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Bootstrap starting.
I20260929 00:59:04.134377 19590 tablet_bootstrap.cc:654] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:04.134584 19589 tablet_bootstrap.cc:654] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:04.135459 19590 log.cc:824] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:04.135875 19591 tablet_bootstrap.cc:654] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:04.135897 19589 log.cc:824] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:04.137209 19591 log.cc:824] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:04.144698 19590 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: No bootstrap required, opened a new log
I20260929 00:59:04.145015 19590 ts_tablet_manager.cc:1397] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260929 00:59:04.145062 19589 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: No bootstrap required, opened a new log
I20260929 00:59:04.145061 19591 tablet_bootstrap.cc:492] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: No bootstrap required, opened a new log
I20260929 00:59:04.145326 19589 ts_tablet_manager.cc:1397] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260929 00:59:04.145323 19591 ts_tablet_manager.cc:1397] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260929 00:59:04.156134 19590 raft_consensus.cc:348] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.156194 19591 raft_consensus.cc:348] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.156558 19590 raft_consensus.cc:374] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:04.156345 19589 raft_consensus.cc:348] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.156661 19591 raft_consensus.cc:374] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:04.156656 19590 raft_consensus.cc:729] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bca7c229a9344e1a4dce70477cc5ccd, State: Initialized, Role: FOLLOWER
I20260929 00:59:04.156755 19591 raft_consensus.cc:729] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff0dfbd1e6746a1b2f444781d750d5e, State: Initialized, Role: FOLLOWER
I20260929 00:59:04.156780 19589 raft_consensus.cc:374] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:04.156870 19589 raft_consensus.cc:729] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e862dbf218b4a879d19f607a8220a31, State: Initialized, Role: FOLLOWER
I20260929 00:59:04.157395 19590 consensus_queue.cc:260] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.157418 19591 consensus_queue.cc:260] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.157444 19589 consensus_queue.cc:260] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.158517 19310 heartbeater.cc:503] Master 127.0.97.62:43287 was elected leader, sending a full tablet report...
I20260929 00:59:04.159194 19590 ts_tablet_manager.cc:1428] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260929 00:59:04.160631 19441 heartbeater.cc:503] Master 127.0.97.62:43287 was elected leader, sending a full tablet report...
I20260929 00:59:04.160920 19589 ts_tablet_manager.cc:1428] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 00:59:04.161132 19591 ts_tablet_manager.cc:1428] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260929 00:59:04.161325 19571 heartbeater.cc:503] Master 127.0.97.62:43287 was elected leader, sending a full tablet report...
I20260929 00:59:04.190791 19597 raft_consensus.cc:482] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:04.191056 19597 raft_consensus.cc:504] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.192731 19597 leader_election.cc:288] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bca7c229a9344e1a4dce70477cc5ccd (127.0.97.1:46341), 8e862dbf218b4a879d19f607a8220a31 (127.0.97.2:35037)
I20260929 00:59:04.198966 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.193292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34257 (local address 127.0.97.2:35037)
0929 00:59:04.193489 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.193497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.193516 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:59:04.193588 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.193592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.193657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.193770 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:04.194260 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.195255 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.197203 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.198146 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.198240 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.198306 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.198627 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.198713 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:59:04.199086 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.192995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35037 (local address 127.0.97.3:34257)
0929 00:59:04.193189 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.193216 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.193344 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.193794 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.193799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.193840 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:04.194067 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.194080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.195389 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.195395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.197039 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.197054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.198292 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.198298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.198468 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.198502 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.198599 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.198737 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:59:04.199710 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.193139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.97.3:50507)
0929 00:59:04.193976 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.193992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:04.194097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:04.194428 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:04.194433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:04.194457 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:04.194813 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.194824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.196015 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.196019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.197595 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:04.197610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.198926 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.198932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:04.199135 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.199176 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 00:59:04.199298 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 00:59:04.199417 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":506,"threads_started":1}
I20260929 00:59:04.199713 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.193206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50507 (local address 127.0.97.1:46341)
0929 00:59:04.193424 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.193433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.194095 (+ 662us) server_negotiation.cc:407] Connection header received
0929 00:59:04.194230 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.194235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.194311 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.194378 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:04.194979 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.195868 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.197783 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.198761 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.198886 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.198975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.199355 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.199486 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:59:04.200212 19396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ce5e7ad1c476db3558a390a6a747e" candidate_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e862dbf218b4a879d19f607a8220a31" is_pre_election: true
I20260929 00:59:04.200803 19396 raft_consensus.cc:2454] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff0dfbd1e6746a1b2f444781d750d5e in term 0.
I20260929 00:59:04.200968 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ce5e7ad1c476db3558a390a6a747e" candidate_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" is_pre_election: true
I20260929 00:59:04.201483 19459 leader_election.cc:302] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e862dbf218b4a879d19f607a8220a31, 9ff0dfbd1e6746a1b2f444781d750d5e; no voters:
I20260929 00:59:04.201550 19265 raft_consensus.cc:2454] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff0dfbd1e6746a1b2f444781d750d5e in term 0.
I20260929 00:59:04.201910 19597 raft_consensus.cc:2781] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:04.202067 19597 raft_consensus.cc:482] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:04.202183 19597 raft_consensus.cc:3037] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:04.204313 19597 raft_consensus.cc:504] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.204952 19597 leader_election.cc:288] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [CANDIDATE]: Term 1 election: Requested vote from peers 7bca7c229a9344e1a4dce70477cc5ccd (127.0.97.1:46341), 8e862dbf218b4a879d19f607a8220a31 (127.0.97.2:35037)
I20260929 00:59:04.205353 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ce5e7ad1c476db3558a390a6a747e" candidate_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bca7c229a9344e1a4dce70477cc5ccd"
I20260929 00:59:04.205485 19396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8ce5e7ad1c476db3558a390a6a747e" candidate_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e862dbf218b4a879d19f607a8220a31"
I20260929 00:59:04.205554 19265 raft_consensus.cc:3037] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:04.205646 19396 raft_consensus.cc:3037] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:04.207734 19265 raft_consensus.cc:2454] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff0dfbd1e6746a1b2f444781d750d5e in term 1.
I20260929 00:59:04.207764 19396 raft_consensus.cc:2454] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff0dfbd1e6746a1b2f444781d750d5e in term 1.
I20260929 00:59:04.208128 19460 leader_election.cc:302] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bca7c229a9344e1a4dce70477cc5ccd, 9ff0dfbd1e6746a1b2f444781d750d5e; no voters:
I20260929 00:59:04.208448 19597 raft_consensus.cc:2781] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:04.208920 19597 raft_consensus.cc:686] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 1 LEADER]: Becoming Leader. State: Replica: 9ff0dfbd1e6746a1b2f444781d750d5e, State: Running, Role: LEADER
I20260929 00:59:04.209372 19597 consensus_queue.cc:237] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260929 00:59:04.213973 19124 catalog_manager.cc:5495] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ff0dfbd1e6746a1b2f444781d750d5e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ff0dfbd1e6746a1b2f444781d750d5e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:04.237102 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.232100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.97.3:36353)
0929 00:59:04.232297 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.232302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.232320 (+ 18us) server_negotiation.cc:407] Connection header received
0929 00:59:04.232442 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.232446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.232584 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.232708 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:04.233042 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.233782 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.235100 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.235503 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.235597 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.236064 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:04.236644 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:04.236784 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.236867 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.236903 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:59:04.245404 19265 raft_consensus.cc:1261] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 1 FOLLOWER]: Refusing update from remote peer 9ff0dfbd1e6746a1b2f444781d750d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:04.245404 19396 raft_consensus.cc:1261] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 1 FOLLOWER]: Refusing update from remote peer 9ff0dfbd1e6746a1b2f444781d750d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:04.246526 19599 consensus_queue.cc:1035] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e862dbf218b4a879d19f607a8220a31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:04.246925 19597 consensus_queue.cc:1035] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bca7c229a9344e1a4dce70477cc5ccd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:04.259125 19603 mvcc.cc:205] Tried to move back new op lower bound from 7334475957218607104 to 7334475957085462528. Current Snapshot: MvccSnapshot[applied={T|T < 7334475957218607104}]
I20260929 00:59:04.259670 19602 mvcc.cc:205] Tried to move back new op lower bound from 7334475957218607104 to 7334475957085462528. Current Snapshot: MvccSnapshot[applied={T|T < 7334475957218607104}]
I20260929 00:59:04.275336 19123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:04.275653 19123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:04.279721 19123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9754770d1e19403c9f9f4da8d1db3b4e: Sending DeleteTablet for 3 replicas of tablet 1b8ce5e7ad1c476db3558a390a6a747e
I20260929 00:59:04.281066 19245 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:04 UTC) from {username='slave'} at 127.0.0.1:40408
I20260929 00:59:04.281162 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:04 UTC) from {username='slave'} at 127.0.0.1:52864
I20260929 00:59:04.281225 19376 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8ce5e7ad1c476db3558a390a6a747e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:04 UTC) from {username='slave'} at 127.0.0.1:39208
I20260929 00:59:04.286499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19183
I20260929 00:59:04.291182 19614 tablet_replica.cc:330] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: stopping tablet replica
I20260929 00:59:04.291651 19614 raft_consensus.cc:2229] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:04.292191 19614 raft_consensus.cc:2258] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:04.294610 19614 ts_tablet_manager.cc:1905] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:04.301088 19612 tablet_replica.cc:330] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: stopping tablet replica
I20260929 00:59:04.301367 19613 tablet_replica.cc:330] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: stopping tablet replica
I20260929 00:59:04.301589 19612 raft_consensus.cc:2229] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:04.301806 19613 raft_consensus.cc:2229] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:04.302155 19612 raft_consensus.cc:2258] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:04.303836 19614 ts_tablet_manager.cc:1918] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:04.304010 19614 log.cc:1196] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/1b8ce5e7ad1c476db3558a390a6a747e
I20260929 00:59:04.304371 19612 ts_tablet_manager.cc:1905] T 1b8ce5e7ad1c476db3558a390a6a747e P 7bca7c229a9344e1a4dce70477cc5ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:04.304600 19614 ts_tablet_manager.cc:1939] T 1b8ce5e7ad1c476db3558a390a6a747e P 8e862dbf218b4a879d19f607a8220a31: Deleting consensus metadata
I20260929 00:59:04.306648 19108 catalog_manager.cc:4841] TS 8e862dbf218b4a879d19f607a8220a31 (127.0.97.2:35037): tablet 1b8ce5e7ad1c476db3558a390a6a747e (table test-table [id=848761be5391408f8866b057a86530b6]) successfully deleted
I20260929 00:59:04.307065 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.288402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.0.97.1:46341)
0929 00:59:04.288646 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.288655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.295592 (+ 6937us) server_negotiation.cc:407] Connection header received
0929 00:59:04.295885 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.295893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.296003 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.296083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:04.296766 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.297992 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.299902 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.300626 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.300745 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.302253 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:04.302263 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:04.302280 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:04.303140 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:04.303203 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:04.303234 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:04.303343 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.306546 (+ 3203us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.306702 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 00:59:04.307835 19613 raft_consensus.cc:2258] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:04.308375 19305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:04.310084 19613 ts_tablet_manager.cc:1905] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:04.317420 19613 ts_tablet_manager.cc:1918] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:04.317587 19613 log.cc:1196] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/1b8ce5e7ad1c476db3558a390a6a747e
I20260929 00:59:04.318130 19613 ts_tablet_manager.cc:1939] T 1b8ce5e7ad1c476db3558a390a6a747e P 9ff0dfbd1e6746a1b2f444781d750d5e: Deleting consensus metadata
I20260929 00:59:04.320032 19108 catalog_manager.cc:4841] TS 9ff0dfbd1e6746a1b2f444781d750d5e (127.0.97.3:36353): tablet 1b8ce5e7ad1c476db3558a390a6a747e (table test-table [id=848761be5391408f8866b057a86530b6]) successfully deleted
I20260929 00:59:04.405560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19183
I20260929 00:59:04.424784 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19314
W20260929 00:59:04.425153 19109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46341 (error 108)
I20260929 00:59:04.427870 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.427312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42742)
0929 00:59:04.427562 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.427695 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 00:59:04.428128 19109 catalog_manager.cc:4584] TS 7bca7c229a9344e1a4dce70477cc5ccd (127.0.97.1:46341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b8ce5e7ad1c476db3558a390a6a747e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
I20260929 00:59:04.433360 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.426821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.97.2:35037)
0929 00:59:04.427003 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.427007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.427024 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:59:04.427079 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.427082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.427160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.427284 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:04.427682 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.428874 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.430258 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.430675 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.430813 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.431928 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:04.431937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:04.431952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:04.432601 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:04.432655 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:04.432674 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:04.432795 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.432922 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.433029 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:59:04.434221 19436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:04.482149 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.481566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42750)
0929 00:59:04.481827 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.481943 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 00:59:04.483340 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.482941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42752)
0929 00:59:04.483102 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.483189 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 00:59:04.543776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19314
I20260929 00:59:04.544399 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.543795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42768)
0929 00:59:04.544055 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.544189 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:59:04.546602 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.546098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42772)
0929 00:59:04.546312 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.546428 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 00:59:04.561347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19444
I20260929 00:59:04.569060 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.562244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.0.97.3:36353)
0929 00:59:04.562441 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:04.562449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:04.562470 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:59:04.562695 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:04.562702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:04.562788 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:04.562904 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:04.563331 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.564123 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.565455 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:04.565894 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:04.566057 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:04.567341 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:04.567352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:04.567366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:04.568140 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:04.568191 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:04.568211 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:04.568346 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:04.568477 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 00:59:04.568689 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:59:04.569787 19566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:04.627208 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.626755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42782)
0929 00:59:04.626967 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.627065 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 00:59:04.628306 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.627921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42784)
0929 00:59:04.628129 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.628195 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 00:59:04.704517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19444
I20260929 00:59:04.723584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19093
I20260929 00:59:04.724797 19154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:04.834281 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:04.833567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46341 (local address 127.0.0.1:42788)
0929 00:59:04.833887 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:04.834020 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 00:59:04.834368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19093
2026-09-29T00:59:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2709 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:04 dist-test-slave-pzsx krb5kdc[19623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:04 dist-test-slave-pzsx krb5kdc[19623](info): set up 2 sockets
Sep 29 00:59:04 dist-test-slave-pzsx krb5kdc[19623](info): commencing operation
krb5kdc: starting...
W20260929 00:59:05.932802 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.009s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:06 dist-test-slave-pzsx krb5kdc[19623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:06.173590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:06.500057 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:06.500370 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:06.500469 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:06.507661 19636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:06.507759 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:06.507804 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:06.507853 19636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:06.507892 19636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:06.517663 19636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:06.518899 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:06.520100 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:06.531857 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:06.531942 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:06.531962 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:06.532649 19636 server_base.cc:1034] running on GCE node
I20260929 00:59:06.533595 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:06.534888 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:06.536083 19636 hybrid_clock.cc:648] HybridClock initialized: now 1790643546536026 us; error 44 us; skew 500 ppm
I20260929 00:59:06.536379 19636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:06 dist-test-slave-pzsx krb5kdc[19623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:06.545365 19636 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:06.546936 19636 webserver.cc:460] Webserver started at http://127.0.97.62:42647/ using document root <none> and password file <none>
I20260929 00:59:06.547605 19636 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:06.547701 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:06.547947 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:06.550985 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "43dbd26f825347258ce32294969ed1dd"
format_stamp: "Formatted at 2026-09-29 00:59:06 on dist-test-slave-pzsx"
server_key: "23bdce30f4850aa965a1465b5f999f1e"
server_key_iv: "dd2af2fc51a880c82f4c4f81d1c50ccd"
server_key_version: "encryptionkey@0"
I20260929 00:59:06.551710 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "43dbd26f825347258ce32294969ed1dd"
format_stamp: "Formatted at 2026-09-29 00:59:06 on dist-test-slave-pzsx"
server_key: "23bdce30f4850aa965a1465b5f999f1e"
server_key_iv: "dd2af2fc51a880c82f4c4f81d1c50ccd"
server_key_version: "encryptionkey@0"
I20260929 00:59:06.556921 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:59:06.560220 19651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:06.561595 19636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:06.561791 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "43dbd26f825347258ce32294969ed1dd"
format_stamp: "Formatted at 2026-09-29 00:59:06 on dist-test-slave-pzsx"
server_key: "23bdce30f4850aa965a1465b5f999f1e"
server_key_iv: "dd2af2fc51a880c82f4c4f81d1c50ccd"
server_key_version: "encryptionkey@0"
I20260929 00:59:06.561988 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:06.597652 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:06.602185 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:06.602494 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:06.621207 19636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36229
I20260929 00:59:06.621245 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36229 every 8 connection(s)
I20260929 00:59:06.622634 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:06.626305 19703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:06.627432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19636
I20260929 00:59:06.627624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:06.628176 388 external_mini_cluster.cc:1428] Setting key 0997e41adeaf20834f8b6c7175b3b534
I20260929 00:59:06.640735 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: Bootstrap starting.
Sep 29 00:59:06 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:06.644304 19703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:06.645433 19703 log.cc:824] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:06.648763 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: No bootstrap required, opened a new log
I20260929 00:59:06.651885 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:06.630060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.0.97.62:36229)
0929 00:59:06.630842 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:06.630854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:06.630896 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:59:06.632684 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:06.632707 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:06.633129 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:06.633603 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:06.635394 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:06.636733 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:06.638577 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:06.639229 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:06.639352 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:06.643946 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:06.643954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:06.643969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:06.644003 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:06.647449 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:06.648117 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:06.648128 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:06.648136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:06.648342 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:06.648772 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:06.648779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:06.648785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:06.649488 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:06.649796 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:06.650111 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 00:59:06.650536 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":363,"threads_started":1}
I20260929 00:59:06.657763 19703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260929 00:59:06.658075 19703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:06.658154 19703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43dbd26f825347258ce32294969ed1dd, State: Initialized, Role: FOLLOWER
I20260929 00:59:06.658596 19703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260929 00:59:06.658787 19703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:06.658882 19703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:06.659000 19703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:06.661365 19703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260929 00:59:06.661841 19703 leader_election.cc:302] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43dbd26f825347258ce32294969ed1dd; no voters:
I20260929 00:59:06.662428 19703 leader_election.cc:288] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:06.662568 19708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:06.663220 19708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [term 1 LEADER]: Becoming Leader. State: Replica: 43dbd26f825347258ce32294969ed1dd, State: Running, Role: LEADER
I20260929 00:59:06.663725 19708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260929 00:59:06.664450 19703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:06.666714 19710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43dbd26f825347258ce32294969ed1dd. Latest consensus state: current_term: 1 leader_uuid: "43dbd26f825347258ce32294969ed1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } } }
I20260929 00:59:06.666774 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43dbd26f825347258ce32294969ed1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbd26f825347258ce32294969ed1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } } }
I20260929 00:59:06.667240 19710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:06.667239 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:06.668315 19714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:06.675166 19714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:06.684677 19714 catalog_manager.cc:1347] Generated new cluster ID: 35ca007402f844e2a6c80c184b6ca6c2
I20260929 00:59:06.684813 19714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:06.706723 19714 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:06.708068 19714 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:06.713598 19714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: Generated new TSK 0
I20260929 00:59:06.714396 19714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 00:59:06.787443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:07.134038 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:07.134443 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:07.134618 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:07.141793 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:07.141917 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:07.142102 19729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:59:07.153051 19729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:07.154333 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:07.155627 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:07.168951 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:07.169013 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:07.169037 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:07.169873 19729 server_base.cc:1034] running on GCE node
I20260929 00:59:07.170406 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:07.171243 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:07.172432 19729 hybrid_clock.cc:648] HybridClock initialized: now 1790643547172378 us; error 51 us; skew 500 ppm
I20260929 00:59:07.172765 19729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:07 dist-test-slave-pzsx krb5kdc[19623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:07.176222 19729 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 00:59:07.177662 19729 webserver.cc:460] Webserver started at http://127.0.97.1:39573/ using document root <none> and password file <none>
I20260929 00:59:07.178354 19729 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:07.178508 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:07.178814 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:07.181749 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "a271b75a8485447fbd33f42369ebac3a"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "428527af63ef6995f31718b7c417b2fd"
server_key_iv: "cfb27a5f30b3e09b9ffb6a06288b333f"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.182529 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a271b75a8485447fbd33f42369ebac3a"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "428527af63ef6995f31718b7c417b2fd"
server_key_iv: "cfb27a5f30b3e09b9ffb6a06288b333f"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.187474 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 00:59:07.190766 19744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.191985 19729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 00:59:07.192165 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "a271b75a8485447fbd33f42369ebac3a"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "428527af63ef6995f31718b7c417b2fd"
server_key_iv: "cfb27a5f30b3e09b9ffb6a06288b333f"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.192350 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:07.212271 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:07.216245 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:07.216468 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:07.217358 19729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:07.218552 19729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:07.218626 19729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.218707 19729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:07.218750 19729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.251164 19729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44027
I20260929 00:59:07.251214 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44027 every 8 connection(s)
I20260929 00:59:07.252430 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:07.255396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19729
I20260929 00:59:07.255599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:07.256047 388 external_mini_cluster.cc:1428] Setting key 68af0d8549c543bfd93d329dee3d98d7
Sep 29 00:59:07 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:07.274739 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:07.255073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40605 (local address 127.0.97.62:36229)
0929 00:59:07.255591 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:07.255597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:07.256683 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 00:59:07.258766 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:07.258772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:07.258855 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:07.258976 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:07.260432 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.261578 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.263074 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.263845 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.263989 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:07.267778 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:07.267787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:07.267795 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:07.267842 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:07.271402 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:07.272095 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:07.272103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:07.272110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:07.272264 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:07.272726 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:07.272733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:07.272740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:07.273109 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:07.273282 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:07.273960 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 00:59:07.274210 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260929 00:59:07.275373 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:07.255617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.1:40605)
0929 00:59:07.256514 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:07.256564 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:07.258475 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:07.259171 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:07.259187 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:07.259704 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:07.260262 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:07.260280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.261727 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.261731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:07.262932 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:07.262944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.264052 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.264057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:07.264309 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:07.265090 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:07.265118 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:07.267550 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:07.271547 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:07.271554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:07.271567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:07.271977 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:07.272374 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:07.272378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:07.272380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:07.272618 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:07.273248 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:07.273253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:07.273549 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 00:59:07.273787 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 00:59:07.274153 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":330,"threads_started":1}
I20260929 00:59:07.277158 19857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260929 00:59:07.277457 19857 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:07.278143 19857 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260929 00:59:07.280660 19668 ts_manager.cc:194] Registered new tserver with Master: a271b75a8485447fbd33f42369ebac3a (127.0.97.1:44027)
I20260929 00:59:07.282882 19668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40605
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 00:59:07.315158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:07.641438 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:07.641763 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:07.641937 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:07.649101 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:07.649240 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:07.649422 19863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:59:07.659082 19863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:07.660382 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:07.661736 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:07.674913 19868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:07.674885 19869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:07.675087 19871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:07.675863 19863 server_base.cc:1034] running on GCE node
I20260929 00:59:07.676573 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:07.677579 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:07.678777 19863 hybrid_clock.cc:648] HybridClock initialized: now 1790643547678729 us; error 43 us; skew 500 ppm
I20260929 00:59:07.679095 19863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:07 dist-test-slave-pzsx krb5kdc[19623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:07.682179 19863 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 00:59:07.683524 19863 webserver.cc:460] Webserver started at http://127.0.97.2:38975/ using document root <none> and password file <none>
I20260929 00:59:07.684198 19863 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:07.684340 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:07.684695 19863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:07.687608 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "f24c4d999e284132a1829c7855d3618c"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "f8b70b32149fbe32c6276ea52b35a890"
server_key_iv: "ea822547737e91a964d6d502440af922"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.688387 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f24c4d999e284132a1829c7855d3618c"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "f8b70b32149fbe32c6276ea52b35a890"
server_key_iv: "ea822547737e91a964d6d502440af922"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.693293 19863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 00:59:07.696745 19878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.697997 19863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:59:07.698166 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "f24c4d999e284132a1829c7855d3618c"
format_stamp: "Formatted at 2026-09-29 00:59:07 on dist-test-slave-pzsx"
server_key: "f8b70b32149fbe32c6276ea52b35a890"
server_key_iv: "ea822547737e91a964d6d502440af922"
server_key_version: "encryptionkey@0"
I20260929 00:59:07.698367 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:07.720717 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:07.725162 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:07.725481 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:07.726411 19863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:07.727697 19863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:07.727800 19863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.727946 19863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:07.728036 19863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:07.760054 19863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44913
I20260929 00:59:07.760111 19990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44913 every 8 connection(s)
I20260929 00:59:07.761389 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:07.769025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19863
I20260929 00:59:07.769212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:07.769644 388 external_mini_cluster.cc:1428] Setting key d29d21183eb59418ec0d448f011f82ba
Sep 29 00:59:07 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:07.784144 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:07.763910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46219 (local address 127.0.97.62:36229)
0929 00:59:07.764079 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:07.764084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:07.765434 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 00:59:07.767375 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:07.767381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:07.767478 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:07.767589 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:07.769347 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.770096 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.772027 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.772450 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.772623 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:07.777122 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:07.777131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:07.777139 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:07.777183 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:07.780837 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:07.781640 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:07.781648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:07.781653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:07.781821 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:07.782234 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:07.782241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:07.782248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:07.782615 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:07.782768 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:07.783427 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 00:59:07.783691 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:59:07.784861 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:07.764385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.2:46219)
0929 00:59:07.765240 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:07.765299 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:07.767103 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:07.767823 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:07.767831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:07.768333 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:07.769142 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:07.769169 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.770280 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.770284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:07.771881 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:07.771898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:07.772741 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:07.772752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:07.773094 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:07.773812 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:07.773844 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:07.776894 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:07.780965 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:07.780976 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:07.780995 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:07.781493 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:07.781931 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:07.781934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:07.781936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:07.782100 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:07.782738 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:07.782743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:07.783046 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 00:59:07.783275 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 00:59:07.783660 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":323,"threads_started":1}
I20260929 00:59:07.786320 19991 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260929 00:59:07.786648 19991 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:07.787334 19991 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260929 00:59:07.788806 19668 ts_manager.cc:194] Registered new tserver with Master: f24c4d999e284132a1829c7855d3618c (127.0.97.2:44913)
I20260929 00:59:07.789589 19668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46219
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 00:59:07.827155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:08.154081 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:08.154364 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:08.154456 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:08.161754 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:08.161865 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:08.161993 19996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:59:08.172045 19996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:08.173345 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:08.174573 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:08.187626 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:08.187567 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:08.187563 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:08.188481 19996 server_base.cc:1034] running on GCE node
I20260929 00:59:08.189142 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:08.190109 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:08.191330 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790643548191286 us; error 56 us; skew 500 ppm
I20260929 00:59:08.191645 19996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:08 dist-test-slave-pzsx krb5kdc[19623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:08.194693 19996 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 00:59:08.196306 19996 webserver.cc:460] Webserver started at http://127.0.97.3:39127/ using document root <none> and password file <none>
I20260929 00:59:08.197189 19996 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:08.197332 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:08.197643 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:08.200590 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "729d6e25d7dd460fa49974f84bc7b6e8"
format_stamp: "Formatted at 2026-09-29 00:59:08 on dist-test-slave-pzsx"
server_key: "05f6d49b02669170db54cc9ff77752cc"
server_key_iv: "d4e478729cdc17b038b557ca00f2a9f2"
server_key_version: "encryptionkey@0"
I20260929 00:59:08.201442 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "729d6e25d7dd460fa49974f84bc7b6e8"
format_stamp: "Formatted at 2026-09-29 00:59:08 on dist-test-slave-pzsx"
server_key: "05f6d49b02669170db54cc9ff77752cc"
server_key_iv: "d4e478729cdc17b038b557ca00f2a9f2"
server_key_version: "encryptionkey@0"
I20260929 00:59:08.206429 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:08.209692 20011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:08.210969 19996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:08.211145 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "729d6e25d7dd460fa49974f84bc7b6e8"
format_stamp: "Formatted at 2026-09-29 00:59:08 on dist-test-slave-pzsx"
server_key: "05f6d49b02669170db54cc9ff77752cc"
server_key_iv: "d4e478729cdc17b038b557ca00f2a9f2"
server_key_version: "encryptionkey@0"
I20260929 00:59:08.211337 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:08.233055 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:08.236927 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:08.237214 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:08.238164 19996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:08.239495 19996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:08.239619 19996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:08.239775 19996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:08.239864 19996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:08.272795 19996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39299
I20260929 00:59:08.272841 20123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39299 every 8 connection(s)
I20260929 00:59:08.274113 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:08.280984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 19996
I20260929 00:59:08.281142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:08.281589 388 external_mini_cluster.cc:1428] Setting key 2fdcfeb1284cbb5af17ee6b5dd5d78e6
I20260929 00:59:08.286199 19857 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
Sep 29 00:59:08 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:08.294369 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.276872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55953 (local address 127.0.97.62:36229)
0929 00:59:08.277046 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.277051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.278282 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 00:59:08.280121 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.280127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.280235 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.280352 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:08.282106 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.283050 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.284687 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.285073 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.285217 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.288860 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:08.288865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:08.288868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:08.288900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:08.291276 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:08.292000 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:08.292010 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:08.292013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:08.292140 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:08.292635 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:08.292640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:08.292642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:08.292906 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:08.293074 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.293776 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.294007 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:59:08.295532 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.277406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.3:55953)
0929 00:59:08.278118 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.278161 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.279849 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.280659 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.280668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.281309 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:08.281930 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.281956 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.283196 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.283203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.284472 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.284483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.285295 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.285305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.285552 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.286223 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:08.286245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:08.288646 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:08.291417 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:08.291426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:08.291443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:08.291870 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:08.292260 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:08.292265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:08.292268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:08.292453 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:08.293035 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:08.293044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:08.293390 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.293654 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.294195 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":290,"threads_started":1}
I20260929 00:59:08.297103 20124 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260929 00:59:08.297379 20124 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:08.298022 20124 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260929 00:59:08.299347 19667 ts_manager.cc:194] Registered new tserver with Master: 729d6e25d7dd460fa49974f84bc7b6e8 (127.0.97.3:39299)
I20260929 00:59:08.300150 19667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55953
I20260929 00:59:08.309548 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:08.325574 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.313512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.0.97.62:36229)
0929 00:59:08.313714 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.313722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.314224 (+ 502us) server_negotiation.cc:407] Connection header received
0929 00:59:08.314333 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.314337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.314399 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.314482 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:08.315876 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.316695 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.317885 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.318275 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.318363 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.320693 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:08.320698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:08.320701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:08.320731 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:08.323125 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:08.323738 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:08.323741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:08.323743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:08.323857 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:08.324152 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:08.324155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:08.324157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:08.324485 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:08.324742 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.325024 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.325216 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:59:08.329608 19667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:08.336664 19667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:08.362948 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.348647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.97.1:44027)
0929 00:59:08.350681 (+ 2034us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.350695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.350731 (+ 36us) server_negotiation.cc:407] Connection header received
0929 00:59:08.350851 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.350863 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.351202 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.351435 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:08.352653 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.354820 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.358525 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.360663 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.360826 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.361163 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.362220 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.362469 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1818,"thread_start_us":1634,"threads_started":1}
I20260929 00:59:08.363142 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.348449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39299 (local address 127.0.0.1:41678)
0929 00:59:08.352394 (+ 3945us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.352410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.352583 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.353541 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.353545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.353572 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:08.354014 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.354030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.356673 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.356679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.359306 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.359328 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.361890 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.361896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.362156 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.362226 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.362337 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.362540 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"mutex_wait_us":4,"thread_start_us":669,"threads_started":1}
I20260929 00:59:08.364492 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.348896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.97.3:39299)
0929 00:59:08.352094 (+ 3198us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.352105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.352716 (+ 611us) server_negotiation.cc:407] Connection header received
0929 00:59:08.352820 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.352827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.353066 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.353261 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:08.354243 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.356436 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.359557 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.361387 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.361511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.361744 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.363710 (+ 1966us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.363960 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3015,"thread_start_us":2873,"threads_started":1}
I20260929 00:59:08.364140 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.348383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44027 (local address 127.0.0.1:38458)
0929 00:59:08.349628 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.349686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.349903 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.351786 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.351799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.351842 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:08.352406 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.352424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.355026 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.355034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.358238 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.358259 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.361357 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.361367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.361664 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.361739 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.361902 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.363601 (+ 1699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":1257,"thread_start_us":3738,"threads_started":1}
I20260929 00:59:08.368826 20059 tablet_service.cc:1467] Processing CreateTablet for tablet 020d3855f56b491baf06e77ff1207632 (DEFAULT_TABLE table=test-table [id=efe8a6c8eedd422a804627ffa349a101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:08.369488 19792 tablet_service.cc:1467] Processing CreateTablet for tablet 020d3855f56b491baf06e77ff1207632 (DEFAULT_TABLE table=test-table [id=efe8a6c8eedd422a804627ffa349a101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:08.370394 20059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d3855f56b491baf06e77ff1207632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:08.370801 19792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d3855f56b491baf06e77ff1207632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:08.373286 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.352582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44913 (local address 127.0.0.1:58726)
0929 00:59:08.364391 (+ 11809us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.364412 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.364535 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.366698 (+ 2163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.366705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.366734 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:08.366897 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.366909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.368842 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.368850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.370643 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.370658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.372452 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.372458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.372689 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.372722 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.372815 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.372962 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11713,"thread_start_us":13240,"threads_started":1}
I20260929 00:59:08.373631 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.352851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.0.97.2:44913)
0929 00:59:08.365738 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.365751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.365786 (+ 35us) server_negotiation.cc:407] Connection header received
0929 00:59:08.365894 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.365902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.366196 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.366401 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:08.367075 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.368677 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.370812 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.372246 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.372375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.372587 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.373114 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.373304 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12721,"thread_start_us":12533,"threads_started":1}
I20260929 00:59:08.377576 19926 tablet_service.cc:1467] Processing CreateTablet for tablet 020d3855f56b491baf06e77ff1207632 (DEFAULT_TABLE table=test-table [id=efe8a6c8eedd422a804627ffa349a101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:08.378993 19926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d3855f56b491baf06e77ff1207632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:08.383116 20142 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Bootstrap starting.
I20260929 00:59:08.386273 20142 tablet_bootstrap.cc:654] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:08.387115 20143 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Bootstrap starting.
I20260929 00:59:08.387609 20142 log.cc:824] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:08.391026 20143 tablet_bootstrap.cc:654] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:08.392119 20142 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: No bootstrap required, opened a new log
I20260929 00:59:08.392253 20143 log.cc:824] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:08.392398 20142 ts_tablet_manager.cc:1397] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 00:59:08.393436 20145 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Bootstrap starting.
I20260929 00:59:08.396811 20145 tablet_bootstrap.cc:654] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:08.396827 20143 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: No bootstrap required, opened a new log
I20260929 00:59:08.397089 20143 ts_tablet_manager.cc:1397] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 00:59:08.397812 20145 log.cc:824] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:08.401155 20145 tablet_bootstrap.cc:492] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: No bootstrap required, opened a new log
I20260929 00:59:08.401423 20145 ts_tablet_manager.cc:1397] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 00:59:08.403367 20142 raft_consensus.cc:348] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.403820 20142 raft_consensus.cc:374] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:08.403923 20142 raft_consensus.cc:729] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d6e25d7dd460fa49974f84bc7b6e8, State: Initialized, Role: FOLLOWER
I20260929 00:59:08.404567 20142 consensus_queue.cc:260] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.405685 20124 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
I20260929 00:59:08.406333 20142 ts_tablet_manager.cc:1428] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 00:59:08.407531 20143 raft_consensus.cc:348] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.407967 20143 raft_consensus.cc:374] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:08.408074 20143 raft_consensus.cc:729] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a271b75a8485447fbd33f42369ebac3a, State: Initialized, Role: FOLLOWER
I20260929 00:59:08.408732 20143 consensus_queue.cc:260] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.410704 20145 raft_consensus.cc:348] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.411237 20145 raft_consensus.cc:374] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:08.411347 20145 raft_consensus.cc:729] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24c4d999e284132a1829c7855d3618c, State: Initialized, Role: FOLLOWER
I20260929 00:59:08.411815 20143 ts_tablet_manager.cc:1428] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 00:59:08.412009 20145 consensus_queue.cc:260] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.417121 19991 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
I20260929 00:59:08.417840 20145 ts_tablet_manager.cc:1428] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
W20260929 00:59:08.504786 19858 tablet.cc:2369] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:59:08.513621 19992 tablet.cc:2369] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:59:08.525799 20125 tablet.cc:2369] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:08.792937 20148 raft_consensus.cc:482] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:08.793216 20148 raft_consensus.cc:504] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.794759 20148 leader_election.cc:288] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a271b75a8485447fbd33f42369ebac3a (127.0.97.1:44027), f24c4d999e284132a1829c7855d3618c (127.0.97.2:44913)
I20260929 00:59:08.801987 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.795218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44027 (local address 127.0.97.3:39633)
0929 00:59:08.795404 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.795425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.795568 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.796123 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.796128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.796156 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:08.796383 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.796393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.797755 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.797759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.799757 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.799772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.801182 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.801186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.801362 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.801393 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.801491 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.801682 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":21}
I20260929 00:59:08.801990 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.795234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39633 (local address 127.0.97.1:44027)
0929 00:59:08.795413 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.795418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.795525 (+ 107us) server_negotiation.cc:407] Connection header received
0929 00:59:08.795719 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.795725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.795803 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.795924 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:08.796523 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.797622 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.799916 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.801039 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.801181 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.801309 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.801533 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.801732 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:59:08.802791 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.795829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53737 (local address 127.0.97.2:44913)
0929 00:59:08.796041 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.796046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.796355 (+ 309us) server_negotiation.cc:407] Connection header received
0929 00:59:08.796533 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.796538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.796619 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.796716 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:08.797159 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.798047 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.800284 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.801547 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.801676 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.801758 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.802370 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.802510 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:59:08.802837 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.795645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44913 (local address 127.0.97.3:53737)
0929 00:59:08.796249 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:08.796265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:08.796385 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:08.796724 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:08.796727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:08.796746 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:08.797017 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.797024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.798194 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.798199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.800139 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:08.800148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.801905 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.801911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:08.802149 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.802189 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 00:59:08.802314 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 00:59:08.802436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":44,"thread_start_us":295,"threads_started":1}
I20260929 00:59:08.803395 19812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d3855f56b491baf06e77ff1207632" candidate_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a271b75a8485447fbd33f42369ebac3a" is_pre_election: true
I20260929 00:59:08.804000 19812 raft_consensus.cc:2454] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d6e25d7dd460fa49974f84bc7b6e8 in term 0.
I20260929 00:59:08.804013 19946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d3855f56b491baf06e77ff1207632" candidate_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24c4d999e284132a1829c7855d3618c" is_pre_election: true
I20260929 00:59:08.804646 19946 raft_consensus.cc:2454] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d6e25d7dd460fa49974f84bc7b6e8 in term 0.
I20260929 00:59:08.804924 20013 leader_election.cc:302] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729d6e25d7dd460fa49974f84bc7b6e8, a271b75a8485447fbd33f42369ebac3a; no voters:
I20260929 00:59:08.805290 20148 raft_consensus.cc:2781] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:08.805409 20148 raft_consensus.cc:482] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:08.805536 20148 raft_consensus.cc:3037] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:08.807660 20148 raft_consensus.cc:504] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.808204 20148 leader_election.cc:288] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [CANDIDATE]: Term 1 election: Requested vote from peers a271b75a8485447fbd33f42369ebac3a (127.0.97.1:44027), f24c4d999e284132a1829c7855d3618c (127.0.97.2:44913)
I20260929 00:59:08.808677 19812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d3855f56b491baf06e77ff1207632" candidate_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a271b75a8485447fbd33f42369ebac3a"
I20260929 00:59:08.808832 19812 raft_consensus.cc:3037] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:08.808727 19946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d3855f56b491baf06e77ff1207632" candidate_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24c4d999e284132a1829c7855d3618c"
I20260929 00:59:08.809047 19946 raft_consensus.cc:3037] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:08.811025 19812 raft_consensus.cc:2454] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d6e25d7dd460fa49974f84bc7b6e8 in term 1.
I20260929 00:59:08.811333 19946 raft_consensus.cc:2454] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d6e25d7dd460fa49974f84bc7b6e8 in term 1.
I20260929 00:59:08.811457 20013 leader_election.cc:302] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729d6e25d7dd460fa49974f84bc7b6e8, a271b75a8485447fbd33f42369ebac3a; no voters:
I20260929 00:59:08.811738 20148 raft_consensus.cc:2781] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:08.812177 20148 raft_consensus.cc:686] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 1 LEADER]: Becoming Leader. State: Replica: 729d6e25d7dd460fa49974f84bc7b6e8, State: Running, Role: LEADER
I20260929 00:59:08.812704 20148 consensus_queue.cc:237] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } }
I20260929 00:59:08.817296 19667 catalog_manager.cc:5495] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 729d6e25d7dd460fa49974f84bc7b6e8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d6e25d7dd460fa49974f84bc7b6e8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20260929 00:59:08.923636 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.918358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.0.97.3:39299)
0929 00:59:08.918862 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.918866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.918958 (+ 92us) server_negotiation.cc:407] Connection header received
0929 00:59:08.919130 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.919134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.919227 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.919367 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:08.919762 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.920415 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.921803 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.922189 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.922283 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.922675 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:08.923158 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:08.923276 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:08.923352 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 00:59:08.923412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":300,"threads_started":1}
I20260929 00:59:08.931116 19946 raft_consensus.cc:1261] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 1 FOLLOWER]: Refusing update from remote peer 729d6e25d7dd460fa49974f84bc7b6e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:08.931231 19812 raft_consensus.cc:1261] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 1 FOLLOWER]: Refusing update from remote peer 729d6e25d7dd460fa49974f84bc7b6e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:08.932061 20152 consensus_queue.cc:1035] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24c4d999e284132a1829c7855d3618c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:08.932403 20148 consensus_queue.cc:1035] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a271b75a8485447fbd33f42369ebac3a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:08.944093 20160 mvcc.cc:205] Tried to move back new op lower bound from 7334475976412717056 to 7334475975940313088. Current Snapshot: MvccSnapshot[applied={T|T < 7334475976412717056}]
I20260929 00:59:08.959326 19667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:08.959594 19667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:08.962747 19667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43dbd26f825347258ce32294969ed1dd: Sending DeleteTablet for 3 replicas of tablet 020d3855f56b491baf06e77ff1207632
I20260929 00:59:08.963807 19792 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d3855f56b491baf06e77ff1207632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38458
I20260929 00:59:08.963855 20059 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d3855f56b491baf06e77ff1207632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41678
I20260929 00:59:08.964224 19926 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d3855f56b491baf06e77ff1207632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58726
I20260929 00:59:08.965849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19729
I20260929 00:59:08.977952 20169 tablet_replica.cc:330] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: stopping tablet replica
I20260929 00:59:08.978505 20169 raft_consensus.cc:2229] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:08.979331 20169 raft_consensus.cc:2258] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:08.983517 20171 tablet_replica.cc:330] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: stopping tablet replica
I20260929 00:59:08.983855 20170 tablet_replica.cc:330] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: stopping tablet replica
I20260929 00:59:08.983958 20171 raft_consensus.cc:2229] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:08.984274 20170 raft_consensus.cc:2229] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:08.984488 20171 raft_consensus.cc:2258] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:08.984788 20170 raft_consensus.cc:2258] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:08.986496 20170 ts_tablet_manager.cc:1905] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:08.987569 20169 ts_tablet_manager.cc:1905] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:08.988708 20171 ts_tablet_manager.cc:1905] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:08.993377 20170 ts_tablet_manager.cc:1918] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:08.993548 20170 log.cc:1196] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-0/wal/wals/020d3855f56b491baf06e77ff1207632
I20260929 00:59:08.994086 20170 ts_tablet_manager.cc:1939] T 020d3855f56b491baf06e77ff1207632 P a271b75a8485447fbd33f42369ebac3a: Deleting consensus metadata
I20260929 00:59:08.995591 20171 ts_tablet_manager.cc:1918] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:08.995733 20171 log.cc:1196] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/020d3855f56b491baf06e77ff1207632
Sep 29 00:59:08 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 00:59:08.996229 20171 ts_tablet_manager.cc:1939] T 020d3855f56b491baf06e77ff1207632 P f24c4d999e284132a1829c7855d3618c: Deleting consensus metadata
I20260929 00:59:08.996177 19653 catalog_manager.cc:4841] TS a271b75a8485447fbd33f42369ebac3a (127.0.97.1:44027): tablet 020d3855f56b491baf06e77ff1207632 (table test-table [id=efe8a6c8eedd422a804627ffa349a101]) successfully deleted
I20260929 00:59:08.997974 19653 catalog_manager.cc:4841] TS f24c4d999e284132a1829c7855d3618c (127.0.97.2:44913): tablet 020d3855f56b491baf06e77ff1207632 (table test-table [id=efe8a6c8eedd422a804627ffa349a101]) successfully deleted
I20260929 00:59:08.998400 20169 ts_tablet_manager.cc:1918] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:08.998587 20169 log.cc:1196] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/020d3855f56b491baf06e77ff1207632
I20260929 00:59:08.999128 20169 ts_tablet_manager.cc:1939] T 020d3855f56b491baf06e77ff1207632 P 729d6e25d7dd460fa49974f84bc7b6e8: Deleting consensus metadata
I20260929 00:59:09.000734 19652 catalog_manager.cc:4841] TS 729d6e25d7dd460fa49974f84bc7b6e8 (127.0.97.3:39299): tablet 020d3855f56b491baf06e77ff1207632 (table test-table [id=efe8a6c8eedd422a804627ffa349a101]) successfully deleted
I20260929 00:59:09.003537 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:08.967439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.0.97.1:44027)
0929 00:59:08.967678 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:08.967687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:08.982640 (+ 14953us) server_negotiation.cc:407] Connection header received
0929 00:59:08.983188 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:08.983196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:08.983290 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:08.983416 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:08.986507 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.987565 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.990485 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:08.991236 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:08.991349 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:08.997410 (+ 6061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:08.997422 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:08.997436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:08.997468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:09.000607 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.001337 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.001347 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.001352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.001529 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.001908 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.001913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.001916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.002350 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:09.002662 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:09.002943 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 00:59:09.003162 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:59:09.004190 19852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:09.094353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19729
I20260929 00:59:09.118752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19863
Sep 29 00:59:09 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 00:59:09.134048 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:09.119656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.0.97.2:44913)
0929 00:59:09.119847 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:09.119852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:09.119874 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:59:09.119944 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:09.119957 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:09.120016 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:09.120131 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:09.121436 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:09.122163 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:09.123420 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:09.123815 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:09.123944 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:09.128235 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:09.128249 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:09.128265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:09.128297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:09.131239 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.132054 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.132062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.132064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.132196 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.132593 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.132599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.132603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.132952 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:09.133266 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:09.133483 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 00:59:09.133695 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:59:09.134747 19986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:09.223385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19863
I20260929 00:59:09.246255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19996
Sep 29 00:59:09 dist-test-slave-pzsx krb5kdc[19623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 00:59:09.264423 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:09.247218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.0.97.3:39299)
0929 00:59:09.247389 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:09.247393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:09.247412 (+ 19us) server_negotiation.cc:407] Connection header received
0929 00:59:09.247466 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:09.247469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:09.247538 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:09.247624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:09.249529 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:09.250350 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:09.252254 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:09.252878 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:09.252984 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:09.258261 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:09.258274 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:09.258293 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:09.258332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:09.261499 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.262194 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.262205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.262213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.262361 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:09.262764 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:09.262769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:09.262772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:09.263155 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:09.263488 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:09.263805 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 00:59:09.264048 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:59:09.265360 20119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:09.385213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19996
I20260929 00:59:09.404670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 19636
I20260929 00:59:09.405725 19698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:09.516261 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 19636
2026-09-29T00:59:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4682 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T00:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:09Z Disabled control of system clock
I20260929 00:59:09.592801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33771
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:59:09.914714 20180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:09.915014 20180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:09.915098 20180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:09.922037 20180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:09.922137 20180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:09.922174 20180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:09.922231 20180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:09.922274 20180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:09.932780 20180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:09.934095 20180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:09.935774 20180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:09.947659 20185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:09.947650 20186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:09.947717 20188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:09.948577 20180 server_base.cc:1034] running on GCE node
I20260929 00:59:09.949368 20180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:09.950752 20180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:09.952003 20180 hybrid_clock.cc:648] HybridClock initialized: now 1790643549951939 us; error 59 us; skew 500 ppm
I20260929 00:59:09.952528 20180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:09.954823 20180 webserver.cc:460] Webserver started at http://127.0.97.62:36675/ using document root <none> and password file <none>
I20260929 00:59:09.955507 20180 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:09.955622 20180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:09.955966 20180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:09.958889 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "5ea64399c38f459db0ededfd1ff20ef0"
format_stamp: "Formatted at 2026-09-29 00:59:09 on dist-test-slave-pzsx"
server_key: "a7f8c0e21fc667c895db055d85c20981"
server_key_iv: "96bce2fe80415569af87a57a66298786"
server_key_version: "encryptionkey@0"
I20260929 00:59:09.959666 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ea64399c38f459db0ededfd1ff20ef0"
format_stamp: "Formatted at 2026-09-29 00:59:09 on dist-test-slave-pzsx"
server_key: "a7f8c0e21fc667c895db055d85c20981"
server_key_iv: "96bce2fe80415569af87a57a66298786"
server_key_version: "encryptionkey@0"
I20260929 00:59:09.964751 20180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 00:59:09.967904 20194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:09.969210 20180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:09.969395 20180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "5ea64399c38f459db0ededfd1ff20ef0"
format_stamp: "Formatted at 2026-09-29 00:59:09 on dist-test-slave-pzsx"
server_key: "a7f8c0e21fc667c895db055d85c20981"
server_key_iv: "96bce2fe80415569af87a57a66298786"
server_key_version: "encryptionkey@0"
I20260929 00:59:09.969640 20180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:09.986349 20180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:09.987107 20180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:09.987375 20180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:10.005306 20180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33771
I20260929 00:59:10.005312 20245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33771 every 8 connection(s)
I20260929 00:59:10.006636 20180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:10.010355 20246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:10.015058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20180
I20260929 00:59:10.015250 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:10.015815 388 external_mini_cluster.cc:1428] Setting key 8dd2eac835ec4de2bff12f77afe823ab
I20260929 00:59:10.021648 20246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: Bootstrap starting.
I20260929 00:59:10.025135 20246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:10.026311 20246 log.cc:824] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:10.029399 20246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: No bootstrap required, opened a new log
I20260929 00:59:10.029589 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:10.017634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.97.62:33771)
0929 00:59:10.018616 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:10.018626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:10.018671 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:59:10.020416 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:10.020438 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:10.020942 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:10.021270 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:10.021869 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.023153 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.025010 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.025737 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.025862 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:10.027134 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:10.027144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:10.027163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:10.027898 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:10.027975 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:10.027996 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:10.028080 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:10.028298 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 00:59:10.028657 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":367,"threads_started":1}
I20260929 00:59:10.040732 20246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260929 00:59:10.041170 20246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:10.041270 20246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea64399c38f459db0ededfd1ff20ef0, State: Initialized, Role: FOLLOWER
I20260929 00:59:10.041885 20246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260929 00:59:10.042116 20246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:10.042244 20246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:10.042420 20246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:10.044605 20246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260929 00:59:10.045106 20246 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea64399c38f459db0ededfd1ff20ef0; no voters:
I20260929 00:59:10.045675 20246 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:10.045847 20251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:10.046531 20251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea64399c38f459db0ededfd1ff20ef0, State: Running, Role: LEADER
I20260929 00:59:10.047058 20251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } }
I20260929 00:59:10.047662 20246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:10.050057 20253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea64399c38f459db0ededfd1ff20ef0. Latest consensus state: current_term: 1 leader_uuid: "5ea64399c38f459db0ededfd1ff20ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } } }
I20260929 00:59:10.050465 20253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:10.050032 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea64399c38f459db0ededfd1ff20ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea64399c38f459db0ededfd1ff20ef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33771 } } }
I20260929 00:59:10.051739 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:10.053416 20257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:10.057817 20257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:10.065768 20257 catalog_manager.cc:1347] Generated new cluster ID: a69e84f092de4b958ba5053866c57d2a
I20260929 00:59:10.065872 20257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:10.084793 20257 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:10.085975 20257 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:10.096014 20257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: Generated new TSK 0
I20260929 00:59:10.096870 20257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:10.102134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 00:59:10.441640 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:10.441959 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:10.442119 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:10.451485 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:10.451639 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:10.451848 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 00:59:10.466145 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:10.467789 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:10.470136 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:10.485404 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:10.485338 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:10.485368 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:10.486483 20270 server_base.cc:1034] running on GCE node
I20260929 00:59:10.487113 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:10.487998 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:10.489197 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790643550489147 us; error 39 us; skew 500 ppm
I20260929 00:59:10.489487 20270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:10.491683 20270 webserver.cc:460] Webserver started at http://127.0.97.1:40267/ using document root <none> and password file <none>
I20260929 00:59:10.492318 20270 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:10.492403 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:10.492763 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:10.495487 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data/instance:
uuid: "f60ad37b366f4eb4be4f69c0f7e973ae"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "cd50529fb21cd04f6f9bad5b4610efdc"
server_key_iv: "4cbdb4e1f98d7d3b0023026e5a981fb1"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.496217 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f60ad37b366f4eb4be4f69c0f7e973ae"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "cd50529fb21cd04f6f9bad5b4610efdc"
server_key_iv: "4cbdb4e1f98d7d3b0023026e5a981fb1"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.501287 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 00:59:10.504566 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.505844 20270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 00:59:10.505971 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal
uuid: "f60ad37b366f4eb4be4f69c0f7e973ae"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "cd50529fb21cd04f6f9bad5b4610efdc"
server_key_iv: "4cbdb4e1f98d7d3b0023026e5a981fb1"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.506151 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:10.522584 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:10.523360 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:10.523588 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:10.524477 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:10.525954 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:10.526043 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.526119 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:10.526182 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.558244 20270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36111
I20260929 00:59:10.558283 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36111 every 8 connection(s)
I20260929 00:59:10.559546 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/data/info.pb
I20260929 00:59:10.559894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20270
I20260929 00:59:10.560082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-0/wal/instance
I20260929 00:59:10.560632 388 external_mini_cluster.cc:1428] Setting key e77a78b59836fa6545b187716c3ac5f6
I20260929 00:59:10.564349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:59:10.572633 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:10.562214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34421 (local address 127.0.97.62:33771)
0929 00:59:10.562657 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:10.562661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:10.563907 (+ 1246us) server_negotiation.cc:407] Connection header received
0929 00:59:10.565745 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:10.565753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:10.565840 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:10.565959 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:10.566972 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.568095 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.569682 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.570393 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.570536 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:10.571811 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:10.571817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:10.571821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:10.571848 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:10.571886 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:10.571898 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:10.571965 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:10.572222 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 00:59:10.572297 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":230,"threads_started":1}
I20260929 00:59:10.574193 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:10.562706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.1:34421)
0929 00:59:10.563715 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:10.563783 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:10.565471 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:10.566148 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:10.566156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:10.566245 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:10.566808 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:10.566824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.568237 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.568240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:10.569539 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:10.569549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:10.570589 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:10.570593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:10.570788 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:10.571496 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:10.571499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:10.571536 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:10.571616 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:10.571995 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:10.572010 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:10.572019 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:59:10.572211 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 00:59:10.572668 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":313,"threads_started":1}
I20260929 00:59:10.575649 20397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260929 00:59:10.575932 20397 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:10.576736 20397 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260929 00:59:10.578693 20211 ts_manager.cc:194] Registered new tserver with Master: f60ad37b366f4eb4be4f69c0f7e973ae (127.0.97.1:36111)
I20260929 00:59:10.580134 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34421
W20260929 00:59:10.886194 20401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:10.886480 20401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:10.886641 20401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:10.893810 20401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:10.893905 20401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:10.894093 20401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 00:59:10.902969 20401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:10.904321 20401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:10.906107 20401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:10.919507 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:10.919576 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:10.919552 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:10.920692 20401 server_base.cc:1034] running on GCE node
I20260929 00:59:10.921295 20401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:10.922241 20401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:10.923463 20401 hybrid_clock.cc:648] HybridClock initialized: now 1790643550923402 us; error 59 us; skew 500 ppm
I20260929 00:59:10.923780 20401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:10.926151 20401 webserver.cc:460] Webserver started at http://127.0.97.2:45233/ using document root <none> and password file <none>
I20260929 00:59:10.927126 20401 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:10.927273 20401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:10.927637 20401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:10.930490 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data/instance:
uuid: "4b16ce378f8748c197882748645848ee"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "b2bd8d90f2c1b644fe84b0a3733805f4"
server_key_iv: "b0eab1068af73551667b0acd10e712c8"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.931208 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4b16ce378f8748c197882748645848ee"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "b2bd8d90f2c1b644fe84b0a3733805f4"
server_key_iv: "b0eab1068af73551667b0acd10e712c8"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.935974 20401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 00:59:10.939188 20415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.940419 20401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 00:59:10.940634 20401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal
uuid: "4b16ce378f8748c197882748645848ee"
format_stamp: "Formatted at 2026-09-29 00:59:10 on dist-test-slave-pzsx"
server_key: "b2bd8d90f2c1b644fe84b0a3733805f4"
server_key_iv: "b0eab1068af73551667b0acd10e712c8"
server_key_version: "encryptionkey@0"
I20260929 00:59:10.940847 20401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:10.958745 20401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:10.959547 20401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:10.959827 20401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:10.960808 20401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:10.962220 20401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:10.962338 20401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.962476 20401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:10.962567 20401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:10.994479 20401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39199
I20260929 00:59:10.994527 20527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39199 every 8 connection(s)
I20260929 00:59:10.995747 20401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/data/info.pb
I20260929 00:59:11.000152 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20401
I20260929 00:59:11.000334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal/instance
I20260929 00:59:11.000972 388 external_mini_cluster.cc:1428] Setting key 9897a7bad8eb9c6ed4ae9a8959122fde
I20260929 00:59:11.004037 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--builtin_ntp_servers=127.0.97.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 00:59:11.010733 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:10.998229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33475 (local address 127.0.97.62:33771)
0929 00:59:10.998411 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:10.998416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:10.999779 (+ 1363us) server_negotiation.cc:407] Connection header received
0929 00:59:11.002244 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.002249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.002321 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.002428 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:11.003782 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.004826 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.007310 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.007938 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.008054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.009800 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:11.009806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:11.009811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:11.009833 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:11.009871 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:11.009886 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:11.009972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.010221 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.010315 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:59:11.012030 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:10.998749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.2:33475)
0929 00:59:10.999598 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:10.999638 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.001965 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.002626 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.002633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.002769 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:11.003567 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.003592 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.004980 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.004989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.007146 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.007162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.008171 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.008189 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.008528 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.009433 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:11.009438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:11.009505 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:11.009616 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:11.009996 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:11.010010 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:11.010021 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.010211 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.010682 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260929 00:59:11.013679 20528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260929 00:59:11.014000 20528 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:11.014724 20528 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260929 00:59:11.016372 20211 ts_manager.cc:194] Registered new tserver with Master: 4b16ce378f8748c197882748645848ee (127.0.97.2:39199)
I20260929 00:59:11.017513 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33475
W20260929 00:59:11.326372 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:11.326718 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:11.326889 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:11.334149 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:11.334290 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:11.334475 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 00:59:11.343708 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:11.345182 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:11.346940 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:11.360651 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:11.360622 20537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:11.360508 20539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:11.361562 20531 server_base.cc:1034] running on GCE node
I20260929 00:59:11.362202 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:11.363140 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:11.364342 20531 hybrid_clock.cc:648] HybridClock initialized: now 1790643551364304 us; error 39 us; skew 500 ppm
I20260929 00:59:11.364715 20531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 00:59:11.366878 20531 webserver.cc:460] Webserver started at http://127.0.97.3:35329/ using document root <none> and password file <none>
I20260929 00:59:11.367558 20531 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:11.367671 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:11.368002 20531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:11.370858 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data/instance:
uuid: "d5b897ed636149239303a5d06cff1ba7"
format_stamp: "Formatted at 2026-09-29 00:59:11 on dist-test-slave-pzsx"
server_key: "80f5c84eb1e6c78b63417e4eb332a067"
server_key_iv: "c65befebf68119835109eaca333f15c4"
server_key_version: "encryptionkey@0"
I20260929 00:59:11.371646 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d5b897ed636149239303a5d06cff1ba7"
format_stamp: "Formatted at 2026-09-29 00:59:11 on dist-test-slave-pzsx"
server_key: "80f5c84eb1e6c78b63417e4eb332a067"
server_key_iv: "c65befebf68119835109eaca333f15c4"
server_key_version: "encryptionkey@0"
I20260929 00:59:11.376603 20531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:59:11.379791 20545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:11.381115 20531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:11.381326 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal
uuid: "d5b897ed636149239303a5d06cff1ba7"
format_stamp: "Formatted at 2026-09-29 00:59:11 on dist-test-slave-pzsx"
server_key: "80f5c84eb1e6c78b63417e4eb332a067"
server_key_iv: "c65befebf68119835109eaca333f15c4"
server_key_version: "encryptionkey@0"
I20260929 00:59:11.381538 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:11.415066 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:11.415946 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:11.416193 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:11.417222 20531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 00:59:11.418694 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 00:59:11.418785 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:11.418870 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 00:59:11.418926 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:11.454510 20531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36597
I20260929 00:59:11.454600 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36597 every 8 connection(s)
I20260929 00:59:11.455849 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/data/info.pb
I20260929 00:59:11.463253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20531
I20260929 00:59:11.463455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal/instance
I20260929 00:59:11.463980 388 external_mini_cluster.cc:1428] Setting key aadfe2649bcceda1496b546499188a4d
I20260929 00:59:11.469077 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.458533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41639 (local address 127.0.97.62:33771)
0929 00:59:11.458717 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.458721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.460142 (+ 1421us) server_negotiation.cc:407] Connection header received
0929 00:59:11.461957 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.461961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.462029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.462124 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:11.463217 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.464343 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.466190 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.466787 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.466927 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.468226 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:11.468231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:11.468234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:11.468251 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:11.468281 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:11.468292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:11.468389 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.468634 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.468727 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:59:11.470074 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.459115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33771 (local address 127.0.97.3:41639)
0929 00:59:11.459987 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.460024 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.461712 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.462322 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.462330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.462423 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:11.463024 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.463040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.464535 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.464541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.466053 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.466069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.467011 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.467018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.467248 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.467892 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 00:59:11.467895 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:11.467942 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 00:59:11.468020 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:11.468387 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:11.468400 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:11.468409 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.468619 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.469018 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260929 00:59:11.471364 20658 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33771
I20260929 00:59:11.471668 20658 heartbeater.cc:464] Registering TS with master...
I20260929 00:59:11.472402 20658 heartbeater.cc:511] Master 127.0.97.62:33771 requested a full tablet report, sending...
I20260929 00:59:11.473821 20211 ts_manager.cc:194] Registered new tserver with Master: d5b897ed636149239303a5d06cff1ba7 (127.0.97.3:36597)
I20260929 00:59:11.474670 20211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41639
I20260929 00:59:11.479095 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 00:59:11.489238 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.483574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.0.97.62:33771)
0929 00:59:11.483743 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.483747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.484215 (+ 468us) server_negotiation.cc:407] Connection header received
0929 00:59:11.484354 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.484358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.484471 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.484616 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:11.485128 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.485897 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.487162 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.487559 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.487654 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.488661 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:11.488665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:11.488668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:11.488687 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:11.488724 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:11.488736 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:11.488806 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.488923 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.488991 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:59:11.493217 20211 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 00:59:11.499914 20211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 00:59:11.522667 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.512552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36597 (local address 127.0.0.1:56222)
0929 00:59:11.513872 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.513892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.514013 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.515451 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.515460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.515492 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:11.515808 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.515817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.517550 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.517558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.519408 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.519420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.521346 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.521355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.521668 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.521754 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.521926 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.522188 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":1279,"threads_started":1}
I20260929 00:59:11.523979 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.513148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.97.3:36597)
0929 00:59:11.514513 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.514519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.514546 (+ 27us) server_negotiation.cc:407] Connection header received
0929 00:59:11.514652 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.514659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.514934 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.515130 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:11.515990 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.517421 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.519586 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.521098 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.521230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.521516 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.523291 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.523532 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1095,"threads_started":1}
I20260929 00:59:11.524530 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.512624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39199 (local address 127.0.0.1:33436)
0929 00:59:11.514071 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.514087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.514183 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.516932 (+ 2749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.516940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.516976 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:11.517436 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.517450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.519834 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.519841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.521770 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.521781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.523754 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.523760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.523923 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.523953 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.524052 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.524189 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":928,"threads_started":1}
I20260929 00:59:11.524628 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.513558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.0.97.2:39199)
0929 00:59:11.516350 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.516358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.516388 (+ 30us) server_negotiation.cc:407] Connection header received
0929 00:59:11.516483 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.516491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.516734 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.516908 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:11.517659 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.519671 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.521953 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.523564 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.523731 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.523972 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.524097 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.524250 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2622,"thread_start_us":2515,"threads_started":1}
I20260929 00:59:11.528129 20593 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b (DEFAULT_TABLE table=test-table [id=5d3aea44efa64a09bdb47b87398a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:11.529717 20593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb4c9c28980436ea4340d8714f9b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:11.531418 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.512104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.0.97.1:36111)
0929 00:59:11.524941 (+ 12837us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.524949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.524978 (+ 29us) server_negotiation.cc:407] Connection header received
0929 00:59:11.525071 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.525078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.525326 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.525780 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:11.526249 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.527242 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.529573 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.530512 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.530621 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.530785 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.531041 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.531157 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12663,"thread_start_us":13351,"threads_started":1}
I20260929 00:59:11.531463 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.511931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53858)
0929 00:59:11.512778 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.512817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.513279 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.525768 (+ 12489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.525773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.525800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:11.526085 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.526094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.527386 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.527391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.529385 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.529400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.530703 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.530709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.530875 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.530908 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.531010 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.531149 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":1055,"threads_started":1}
I20260929 00:59:11.535200 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b (DEFAULT_TABLE table=test-table [id=5d3aea44efa64a09bdb47b87398a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:11.535200 20463 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b (DEFAULT_TABLE table=test-table [id=5d3aea44efa64a09bdb47b87398a6bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 00:59:11.536665 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb4c9c28980436ea4340d8714f9b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:11.536661 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb4c9c28980436ea4340d8714f9b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:11.541416 20676 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Bootstrap starting.
I20260929 00:59:11.545433 20676 tablet_bootstrap.cc:654] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:11.546520 20676 log.cc:824] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:11.549787 20676 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: No bootstrap required, opened a new log
I20260929 00:59:11.550071 20676 ts_tablet_manager.cc:1397] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 00:59:11.553836 20679 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Bootstrap starting.
I20260929 00:59:11.554131 20678 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Bootstrap starting.
I20260929 00:59:11.557279 20678 tablet_bootstrap.cc:654] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:11.557500 20679 tablet_bootstrap.cc:654] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:11.558641 20678 log.cc:824] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:11.558846 20679 log.cc:824] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:11.561030 20676 raft_consensus.cc:348] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.561772 20676 raft_consensus.cc:374] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:11.561954 20676 raft_consensus.cc:729] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b897ed636149239303a5d06cff1ba7, State: Initialized, Role: FOLLOWER
I20260929 00:59:11.562988 20676 consensus_queue.cc:260] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.567143 20676 ts_tablet_manager.cc:1428] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260929 00:59:11.568003 20658 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260929 00:59:11.571561 20678 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: No bootstrap required, opened a new log
I20260929 00:59:11.571949 20678 ts_tablet_manager.cc:1397] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20260929 00:59:11.575388 20679 tablet_bootstrap.cc:492] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: No bootstrap required, opened a new log
I20260929 00:59:11.575700 20679 ts_tablet_manager.cc:1397] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20260929 00:59:11.581432 20678 raft_consensus.cc:348] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.581892 20678 raft_consensus.cc:374] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:11.581993 20678 raft_consensus.cc:729] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60ad37b366f4eb4be4f69c0f7e973ae, State: Initialized, Role: FOLLOWER
I20260929 00:59:11.582581 20678 consensus_queue.cc:260] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.582957 20397 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260929 00:59:11.583419 20679 raft_consensus.cc:348] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.583777 20679 raft_consensus.cc:374] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:11.583840 20679 raft_consensus.cc:729] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b16ce378f8748c197882748645848ee, State: Initialized, Role: FOLLOWER
I20260929 00:59:11.584370 20678 ts_tablet_manager.cc:1428] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260929 00:59:11.584424 20679 consensus_queue.cc:260] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.585585 20528 heartbeater.cc:503] Master 127.0.97.62:33771 was elected leader, sending a full tablet report...
I20260929 00:59:11.586278 20679 ts_tablet_manager.cc:1428] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 00:59:11.676353 20682 raft_consensus.cc:482] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 00:59:11.676748 20682 raft_consensus.cc:504] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.678490 20682 leader_election.cc:288] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f60ad37b366f4eb4be4f69c0f7e973ae (127.0.97.1:36111), 4b16ce378f8748c197882748645848ee (127.0.97.2:39199)
I20260929 00:59:11.685020 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.678781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39199 (local address 127.0.97.3:46251)
0929 00:59:11.679075 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.679097 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.679241 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.679665 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.679670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.679708 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:11.679995 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.680006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.681397 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.681403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.683038 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.683049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.684233 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.684240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.684441 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.684480 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.684608 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.684763 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":47,"spinlock_wait_cycles":23808}
I20260929 00:59:11.685056 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.678915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46251 (local address 127.0.97.2:39199)
0929 00:59:11.679101 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.679108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.679193 (+ 85us) server_negotiation.cc:407] Connection header received
0929 00:59:11.679409 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.679413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.679484 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.679590 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:11.680180 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.681260 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.683192 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.684084 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.684180 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.684248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.684661 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.684793 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:11.685555 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.678950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.97.3:56791)
0929 00:59:11.679701 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.679714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:11.679828 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:11.680180 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:11.680184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:11.680207 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 00:59:11.680612 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.680625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.681735 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.681739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.683375 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:11.683389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.684756 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.684761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:11.684934 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.684965 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 00:59:11.685122 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 00:59:11.685233 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":19,"thread_start_us":445,"threads_started":1}
I20260929 00:59:11.685693 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.679075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56791 (local address 127.0.97.1:36111)
0929 00:59:11.679289 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.679297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.679798 (+ 501us) server_negotiation.cc:407] Connection header received
0929 00:59:11.679953 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.679958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.680042 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.680134 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 00:59:11.680765 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.681583 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.683552 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.684557 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.684679 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.684774 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.685291 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.685430 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 00:59:11.686264 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb4c9c28980436ea4340d8714f9b18b" candidate_uuid: "d5b897ed636149239303a5d06cff1ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b16ce378f8748c197882748645848ee" is_pre_election: true
I20260929 00:59:11.686700 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb4c9c28980436ea4340d8714f9b18b" candidate_uuid: "d5b897ed636149239303a5d06cff1ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" is_pre_election: true
I20260929 00:59:11.686900 20483 raft_consensus.cc:2454] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b897ed636149239303a5d06cff1ba7 in term 0.
I20260929 00:59:11.687263 20352 raft_consensus.cc:2454] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b897ed636149239303a5d06cff1ba7 in term 0.
I20260929 00:59:11.687601 20546 leader_election.cc:302] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b16ce378f8748c197882748645848ee, d5b897ed636149239303a5d06cff1ba7; no voters:
I20260929 00:59:11.688021 20682 raft_consensus.cc:2781] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 00:59:11.688169 20682 raft_consensus.cc:482] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 00:59:11.688277 20682 raft_consensus.cc:3037] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:11.690478 20682 raft_consensus.cc:504] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.691056 20682 leader_election.cc:288] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [CANDIDATE]: Term 1 election: Requested vote from peers f60ad37b366f4eb4be4f69c0f7e973ae (127.0.97.1:36111), 4b16ce378f8748c197882748645848ee (127.0.97.2:39199)
I20260929 00:59:11.691448 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb4c9c28980436ea4340d8714f9b18b" candidate_uuid: "d5b897ed636149239303a5d06cff1ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae"
I20260929 00:59:11.691498 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb4c9c28980436ea4340d8714f9b18b" candidate_uuid: "d5b897ed636149239303a5d06cff1ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b16ce378f8748c197882748645848ee"
I20260929 00:59:11.691656 20352 raft_consensus.cc:3037] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:11.691677 20483 raft_consensus.cc:3037] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:11.694923 20483 raft_consensus.cc:2454] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b897ed636149239303a5d06cff1ba7 in term 1.
I20260929 00:59:11.694921 20352 raft_consensus.cc:2454] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b897ed636149239303a5d06cff1ba7 in term 1.
I20260929 00:59:11.695343 20546 leader_election.cc:302] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b16ce378f8748c197882748645848ee, d5b897ed636149239303a5d06cff1ba7, f60ad37b366f4eb4be4f69c0f7e973ae; no voters:
I20260929 00:59:11.695675 20682 raft_consensus.cc:2781] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:11.696102 20682 raft_consensus.cc:686] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 1 LEADER]: Becoming Leader. State: Replica: d5b897ed636149239303a5d06cff1ba7, State: Running, Role: LEADER
I20260929 00:59:11.696619 20682 consensus_queue.cc:237] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } }
I20260929 00:59:11.700966 20210 catalog_manager.cc:5495] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b897ed636149239303a5d06cff1ba7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d5b897ed636149239303a5d06cff1ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b897ed636149239303a5d06cff1ba7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } health_report { overall_health: HEALTHY } } }
W20260929 00:59:11.707511 20659 tablet.cc:2369] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:11.715728 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.710780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.0.97.3:36597)
0929 00:59:11.711005 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.711010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.711069 (+ 59us) server_negotiation.cc:407] Connection header received
0929 00:59:11.711233 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.711237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.711318 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.711434 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:11.711728 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.712411 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.713737 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.714148 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.714236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.714680 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:11.715175 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:11.715343 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.715399 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.715476 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:59:11.723598 20483 raft_consensus.cc:1261] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 1 FOLLOWER]: Refusing update from remote peer d5b897ed636149239303a5d06cff1ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:11.723624 20352 raft_consensus.cc:1261] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 1 FOLLOWER]: Refusing update from remote peer d5b897ed636149239303a5d06cff1ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:11.724601 20686 consensus_queue.cc:1035] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b16ce378f8748c197882748645848ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:11.724947 20682 consensus_queue.cc:1035] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60ad37b366f4eb4be4f69c0f7e973ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:11.738663 20690 mvcc.cc:205] Tried to move back new op lower bound from 7334475987850268672 to 7334475987753209856. Current Snapshot: MvccSnapshot[applied={T|T < 7334475987850268672}]
I20260929 00:59:11.751770 20211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:11.752033 20211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 00:59:11.755563 20211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ea64399c38f459db0ededfd1ff20ef0: Sending DeleteTablet for 3 replicas of tablet 9bb4c9c28980436ea4340d8714f9b18b
I20260929 00:59:11.756800 20463 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:11 UTC) from {username='slave'} at 127.0.0.1:33436
I20260929 00:59:11.756798 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:11 UTC) from {username='slave'} at 127.0.0.1:53858
I20260929 00:59:11.757030 20593 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb4c9c28980436ea4340d8714f9b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 00:59:11 UTC) from {username='slave'} at 127.0.0.1:56222
I20260929 00:59:11.758347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20270
I20260929 00:59:11.767800 20699 tablet_replica.cc:330] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: stopping tablet replica
I20260929 00:59:11.769255 20699 raft_consensus.cc:2229] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:11.770684 20699 raft_consensus.cc:2258] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:11.775161 20699 ts_tablet_manager.cc:1905] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:11.777886 20700 tablet_replica.cc:330] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: stopping tablet replica
I20260929 00:59:11.778380 20700 raft_consensus.cc:2229] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 00:59:11.779095 20700 raft_consensus.cc:2258] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:11.781035 20698 tablet_replica.cc:330] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: stopping tablet replica
I20260929 00:59:11.781230 20700 ts_tablet_manager.cc:1905] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:11.781594 20698 raft_consensus.cc:2229] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 00:59:11.782145 20698 raft_consensus.cc:2258] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 00:59:11.784054 20698 ts_tablet_manager.cc:1905] T 9bb4c9c28980436ea4340d8714f9b18b P f60ad37b366f4eb4be4f69c0f7e973ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 00:59:11.784557 20699 ts_tablet_manager.cc:1918] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:11.784736 20699 log.cc:1196] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-1/wal/wals/9bb4c9c28980436ea4340d8714f9b18b
I20260929 00:59:11.785368 20699 ts_tablet_manager.cc:1939] T 9bb4c9c28980436ea4340d8714f9b18b P 4b16ce378f8748c197882748645848ee: Deleting consensus metadata
I20260929 00:59:11.786206 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.760027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.0.97.1:36111)
0929 00:59:11.760281 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.760290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.770523 (+ 10233us) server_negotiation.cc:407] Connection header received
0929 00:59:11.771075 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.771085 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.771268 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.771375 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:11.772157 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.773385 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.775594 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.776263 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.776379 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.783777 (+ 7398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:11.783788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:11.783805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:11.784626 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:11.784707 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:11.784737 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:11.784830 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.785667 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.785801 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:11.786981 20392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:11.787384 20195 catalog_manager.cc:4841] TS 4b16ce378f8748c197882748645848ee (127.0.97.2:39199): tablet 9bb4c9c28980436ea4340d8714f9b18b (table test-table [id=5d3aea44efa64a09bdb47b87398a6bb1]) successfully deleted
I20260929 00:59:11.792673 20700 ts_tablet_manager.cc:1918] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 00:59:11.792963 20700 log.cc:1196] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Deleting WAL directory at /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643237839665-388-0/minicluster-data/ts-2/wal/wals/9bb4c9c28980436ea4340d8714f9b18b
I20260929 00:59:11.793509 20700 ts_tablet_manager.cc:1939] T 9bb4c9c28980436ea4340d8714f9b18b P d5b897ed636149239303a5d06cff1ba7: Deleting consensus metadata
I20260929 00:59:11.795356 20197 catalog_manager.cc:4841] TS d5b897ed636149239303a5d06cff1ba7 (127.0.97.3:36597): tablet 9bb4c9c28980436ea4340d8714f9b18b (table test-table [id=5d3aea44efa64a09bdb47b87398a6bb1]) successfully deleted
I20260929 00:59:11.899204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20270
I20260929 00:59:11.916765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20401
W20260929 00:59:11.916826 20198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36111 (error 108)
I20260929 00:59:11.919081 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.918550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53870)
0929 00:59:11.918797 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.918925 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 00:59:11.919348 20198 catalog_manager.cc:4584] TS f60ad37b366f4eb4be4f69c0f7e973ae (127.0.97.1:36111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bb4c9c28980436ea4340d8714f9b18b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
I20260929 00:59:11.924968 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.917797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.97.2:39199)
0929 00:59:11.917979 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:11.917985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:11.918005 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:59:11.918215 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:11.918221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:11.918331 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:11.918443 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:11.918924 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.920063 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.921436 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:11.922075 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:11.922204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:11.923224 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:11.923235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:11.923253 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:11.924035 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:11.924093 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:11.924116 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:11.924227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:11.924353 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 00:59:11.924464 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:59:11.925683 20523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:11.973053 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.972478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53884)
0929 00:59:11.972722 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.972827 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 00:59:11.974200 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:11.973785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53890)
0929 00:59:11.973938 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:11.974054 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 00:59:12.020875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20401
I20260929 00:59:12.043218 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.042569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53896)
0929 00:59:12.042921 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:12.043032 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260929 00:59:12.044554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20531
I20260929 00:59:12.044881 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.044297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53904)
0929 00:59:12.044642 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:12.044731 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260929 00:59:12.052459 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.045540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.0.97.3:36597)
0929 00:59:12.045697 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:12.045701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:12.045716 (+ 15us) server_negotiation.cc:407] Connection header received
0929 00:59:12.045767 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:12.045771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:12.045851 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:12.045933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:12.046463 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:12.047272 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:12.049224 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:12.049679 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:12.049783 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:12.050958 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:12.050970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:12.050988 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:12.051604 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:12.051653 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:12.051674 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:12.051761 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:12.051976 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 00:59:12.052093 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:12.053233 20653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:12.125705 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.125101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53920)
0929 00:59:12.125376 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:12.125490 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 00:59:12.126883 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.126424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53924)
0929 00:59:12.126662 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:12.126750 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 00:59:12.175279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20531
I20260929 00:59:12.194690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20180
I20260929 00:59:12.195809 20241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:12.303639 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:12.302990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36111 (local address 127.0.0.1:53936)
0929 00:59:12.303275 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:12.303433 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 00:59:12.303598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20180
2026-09-29T00:59:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2791 ms)
[----------] 6 tests from EncryptionPolicyTest (21441 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:12 dist-test-slave-pzsx krb5kdc[20709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:12 dist-test-slave-pzsx krb5kdc[20709](info): set up 2 sockets
Sep 29 00:59:12 dist-test-slave-pzsx krb5kdc[20709](info): commencing operation
krb5kdc: starting...
W20260929 00:59:13.401036 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:13 dist-test-slave-pzsx krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:13.648895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:13.974849 20722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:13.975109 20722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:13.975278 20722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:13.982698 20722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:13.982832 20722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:13.982918 20722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:13.983004 20722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:13.983068 20722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:13.997084 20722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:13.998376 20722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:13.999644 20722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:14.011252 20727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.011313 20728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:14.011332 20730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:14.012358 20722 server_base.cc:1034] running on GCE node
I20260929 00:59:14.013176 20722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:14.014549 20722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:14.015739 20722 hybrid_clock.cc:648] HybridClock initialized: now 1790643554015686 us; error 54 us; skew 500 ppm
I20260929 00:59:14.016031 20722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:14 dist-test-slave-pzsx krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:14.019164 20722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:14.020582 20722 webserver.cc:460] Webserver started at http://127.0.0.1:45269/ using document root <none> and password file <none>
I20260929 00:59:14.021219 20722 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:14.021315 20722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:14.021560 20722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:14.024418 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "c6053283a5ab4f74918cba69ba25fdbf"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-pzsx"
server_key: "4c4c89e0ebdf77cbb6467470a013065d"
server_key_iv: "0f2b5c49577435d6e63944138ae04a2f"
server_key_version: "encryptionkey@0"
I20260929 00:59:14.025189 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6053283a5ab4f74918cba69ba25fdbf"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-pzsx"
server_key: "4c4c89e0ebdf77cbb6467470a013065d"
server_key_iv: "0f2b5c49577435d6e63944138ae04a2f"
server_key_version: "encryptionkey@0"
I20260929 00:59:14.030103 20722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:14.034488 20737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:14.036367 20722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 00:59:14.036563 20722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "c6053283a5ab4f74918cba69ba25fdbf"
format_stamp: "Formatted at 2026-09-29 00:59:14 on dist-test-slave-pzsx"
server_key: "4c4c89e0ebdf77cbb6467470a013065d"
server_key_iv: "0f2b5c49577435d6e63944138ae04a2f"
server_key_version: "encryptionkey@0"
I20260929 00:59:14.036796 20722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:14.051167 20722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:14.055002 20722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:14.055269 20722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:14.073675 20722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36697
I20260929 00:59:14.073704 20788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36697 every 8 connection(s)
I20260929 00:59:14.075070 20722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:14.078713 20789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:14.083364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20722
I20260929 00:59:14.083609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:14.084254 388 external_mini_cluster.cc:1428] Setting key 6666a3cac1f55de19c6c5e5a8a392c77
I20260929 00:59:14.090644 20789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf: Bootstrap starting.
I20260929 00:59:14.094247 20789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:14.095247 20789 log.cc:824] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:14.098074 20789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf: No bootstrap required, opened a new log
Sep 29 00:59:14 dist-test-slave-pzsx krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:59:14.108114 20789 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260929 00:59:14.108608 20789 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:14.108726 20789 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6053283a5ab4f74918cba69ba25fdbf, State: Initialized, Role: FOLLOWER
I20260929 00:59:14.109292 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.086294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51537 (local address 127.0.0.1:36697)
0929 00:59:14.087175 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:14.087203 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:14.087257 (+ 54us) server_negotiation.cc:407] Connection header received
0929 00:59:14.089665 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:14.089693 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:14.090166 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:14.090598 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:14.091975 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.093943 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.095297 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.096247 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.096384 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.100589 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:14.100603 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:14.100622 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:14.100671 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:14.104674 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.105494 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.105503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.105518 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.105735 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.106232 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.106236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.106240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.106948 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:14.107292 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:14.107577 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 00:59:14.107905 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":373,"threads_started":1}
I20260929 00:59:14.109423 20789 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260929 00:59:14.109756 20789 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:14.109910 20789 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:14.110091 20789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:14.112627 20789 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260929 00:59:14.113128 20789 leader_election.cc:302] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6053283a5ab4f74918cba69ba25fdbf; no voters:
I20260929 00:59:14.113436 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:14.113792 20789 leader_election.cc:288] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:14.113893 20794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:14.114557 20794 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [term 1 LEADER]: Becoming Leader. State: Replica: c6053283a5ab4f74918cba69ba25fdbf, State: Running, Role: LEADER
I20260929 00:59:14.115187 20794 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260929 00:59:14.116935 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6053283a5ab4f74918cba69ba25fdbf. Latest consensus state: current_term: 1 leader_uuid: "c6053283a5ab4f74918cba69ba25fdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } } }
I20260929 00:59:14.117301 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:14.117722 20796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6053283a5ab4f74918cba69ba25fdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6053283a5ab4f74918cba69ba25fdbf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } } }
I20260929 00:59:14.118116 20796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:14.131160 20789 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:14.131338 20798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:14.137991 20798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:14.153859 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:14.133558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35697 (local address 127.0.0.1:36697)
0929 00:59:14.133791 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:14.133801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:14.134387 (+ 586us) server_negotiation.cc:407] Connection header received
0929 00:59:14.134664 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:14.134670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:14.134750 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:14.134830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:14.137065 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.138280 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.140174 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:14.140934 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:14.141099 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:14.144359 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:14.144366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:14.144371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:14.144409 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:14.148457 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.149892 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.149901 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.149907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.150137 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:14.150852 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:14.150860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:14.150866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:14.151409 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:14.151716 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:14.152633 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 00:59:14.152881 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:59:14.163995 20798 catalog_manager.cc:1347] Generated new cluster ID: 05e3812190b345c2ae117baa790c1235
I20260929 00:59:14.164147 20798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:14.200840 20798 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:14.202281 20798 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:14.208266 20798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6053283a5ab4f74918cba69ba25fdbf: Generated new TSK 0
I20260929 00:59:14.209357 20798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:14.227844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20722
I20260929 00:59:14.229152 20784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:14.324811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20722
W20260929 00:59:14.342357 20707 connection.cc:535] client connection to 127.0.0.1:36697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36697): Connection reset by peer (error 104)
2026-09-29T00:59:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:14 dist-test-slave-pzsx krb5kdc[20821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:14 dist-test-slave-pzsx krb5kdc[20821](info): set up 2 sockets
Sep 29 00:59:14 dist-test-slave-pzsx krb5kdc[20821](info): commencing operation
krb5kdc: starting...
W20260929 00:59:15.408746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:15 dist-test-slave-pzsx krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:15.654099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:15.981477 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:15.981830 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:15.981992 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:15.989451 20834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:15.989626 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:15.989743 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:15.989852 20834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:15.989938 20834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:15.999768 20834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:16.001092 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:16.002382 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:16.014055 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:16.014083 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:16.014138 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:16.015065 20834 server_base.cc:1034] running on GCE node
I20260929 00:59:16.015848 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:16.017333 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:16.018563 20834 hybrid_clock.cc:648] HybridClock initialized: now 1790643556018503 us; error 70 us; skew 500 ppm
I20260929 00:59:16.018913 20834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:16 dist-test-slave-pzsx krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:16.022028 20834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:16.023473 20834 webserver.cc:460] Webserver started at http://127.0.0.1:35845/ using document root <none> and password file <none>
I20260929 00:59:16.024137 20834 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:16.024288 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:16.024680 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:16.027699 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "77c82eb9567744039b94ff58324221f4"
format_stamp: "Formatted at 2026-09-29 00:59:16 on dist-test-slave-pzsx"
server_key: "95682f5e656ce3df2b40d787457c8057"
server_key_iv: "ce9fce2d65bffde6a2c984f9c4e8d7de"
server_key_version: "encryptionkey@0"
I20260929 00:59:16.028879 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "77c82eb9567744039b94ff58324221f4"
format_stamp: "Formatted at 2026-09-29 00:59:16 on dist-test-slave-pzsx"
server_key: "95682f5e656ce3df2b40d787457c8057"
server_key_iv: "ce9fce2d65bffde6a2c984f9c4e8d7de"
server_key_version: "encryptionkey@0"
I20260929 00:59:16.036406 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260929 00:59:16.041117 20849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:16.042434 20834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:16.042626 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "77c82eb9567744039b94ff58324221f4"
format_stamp: "Formatted at 2026-09-29 00:59:16 on dist-test-slave-pzsx"
server_key: "95682f5e656ce3df2b40d787457c8057"
server_key_iv: "ce9fce2d65bffde6a2c984f9c4e8d7de"
server_key_version: "encryptionkey@0"
I20260929 00:59:16.042837 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:16.062739 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:16.066581 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:16.066816 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:16.084640 20834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36927
I20260929 00:59:16.084722 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36927 every 8 connection(s)
I20260929 00:59:16.085950 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:16.088001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20834
I20260929 00:59:16.088184 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:16.088637 388 external_mini_cluster.cc:1428] Setting key bf4205744f46c9f5016afdad6f56aa7d
I20260929 00:59:16.090018 20901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:16.103006 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.090387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57947 (local address 127.0.0.1:36927)
0929 00:59:16.091179 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:16.091193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:16.091242 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:59:16.093251 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:16.093278 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:16.093790 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:16.094213 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:16.094718 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.096682 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.097966 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.098944 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.099112 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.100267 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:16.100277 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:16.100300 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:16.101067 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:16.101156 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:16.101191 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:16.101368 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:16.101572 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 00:59:16.101741 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":319,"threads_started":1}
I20260929 00:59:16.104473 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4: Bootstrap starting.
I20260929 00:59:16.106891 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:16.108114 20901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:16.109323 20901 log.cc:824] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:16.112430 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4: No bootstrap required, opened a new log
I20260929 00:59:16.119236 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:16.112126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32771 (local address 127.0.0.1:36927)
0929 00:59:16.112350 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:16.112358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:16.112906 (+ 548us) server_negotiation.cc:407] Connection header received
0929 00:59:16.113056 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:16.113063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:16.113135 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:16.113248 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:16.113727 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.114989 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.116425 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:16.117114 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:16.117253 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:16.118361 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:16.118367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:16.118372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:16.118393 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:16.118443 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:16.118463 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:16.118602 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:16.118825 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:59:16.118884 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:59:16.122310 20901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 00:59:16.122665 20901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:16.122738 20901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c82eb9567744039b94ff58324221f4, State: Initialized, Role: FOLLOWER
I20260929 00:59:16.123302 20901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 00:59:16.123513 20901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:16.123610 20901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:16.123737 20901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:16.126088 20901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 00:59:16.126536 20901 leader_election.cc:302] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c82eb9567744039b94ff58324221f4; no voters:
I20260929 00:59:16.127039 20901 leader_election.cc:288] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:16.127187 20911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:16.127894 20911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [term 1 LEADER]: Becoming Leader. State: Replica: 77c82eb9567744039b94ff58324221f4, State: Running, Role: LEADER
I20260929 00:59:16.128407 20911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 00:59:16.129161 20901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:16.131270 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c82eb9567744039b94ff58324221f4. Latest consensus state: current_term: 1 leader_uuid: "77c82eb9567744039b94ff58324221f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260929 00:59:16.131724 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:16.131270 20912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c82eb9567744039b94ff58324221f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c82eb9567744039b94ff58324221f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260929 00:59:16.132870 20912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:16.132896 20916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:16.138604 20916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:16.147945 20916 catalog_manager.cc:1347] Generated new cluster ID: 11a244472c2a45f58bb9c9cf758cb635
I20260929 00:59:16.148099 20916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:16.158907 20916 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:16.160175 20916 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:16.176442 20916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c82eb9567744039b94ff58324221f4: Generated new TSK 0
I20260929 00:59:16.177323 20916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:16.192590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20834
W20260929 00:59:16.193742 20896 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57947
W20260929 00:59:16.194476 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20834: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:16.194659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20834
W20260929 00:59:16.211462 20819 connection.cc:535] client connection to 127.0.0.1:36927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36927): Connection reset by peer (error 104)
2026-09-29T00:59:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:16 dist-test-slave-pzsx krb5kdc[20932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:16 dist-test-slave-pzsx krb5kdc[20932](info): set up 2 sockets
Sep 29 00:59:16 dist-test-slave-pzsx krb5kdc[20932](info): commencing operation
krb5kdc: starting...
W20260929 00:59:17.288993 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:17 dist-test-slave-pzsx krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:17.547163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:17.876631 20945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:17.876978 20945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:17.877086 20945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:17.884246 20945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:17.884363 20945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:17.884410 20945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:17.884473 20945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:17.884578 20945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:17.894501 20945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:17.895759 20945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:17.897045 20945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:17.908926 20950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:17.908963 20953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:17.908998 20951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:17.910270 20945 server_base.cc:1034] running on GCE node
I20260929 00:59:17.911090 20945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:17.912472 20945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:17.913843 20945 hybrid_clock.cc:648] HybridClock initialized: now 1790643557913797 us; error 37 us; skew 500 ppm
I20260929 00:59:17.914248 20945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:17 dist-test-slave-pzsx krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:17.923682 20945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:17.925457 20945 webserver.cc:460] Webserver started at http://127.0.0.1:39895/ using document root <none> and password file <none>
I20260929 00:59:17.926188 20945 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:17.926357 20945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:17.926692 20945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:17.929807 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "a80249ac31964c54bc77bcb6a2b120dd"
format_stamp: "Formatted at 2026-09-29 00:59:17 on dist-test-slave-pzsx"
server_key: "d95ca1798e96a3baa3a3a26021ccf128"
server_key_iv: "1e9424a7b4d944a47c5f3a39c3e33fe8"
server_key_version: "encryptionkey@0"
I20260929 00:59:17.930603 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "a80249ac31964c54bc77bcb6a2b120dd"
format_stamp: "Formatted at 2026-09-29 00:59:17 on dist-test-slave-pzsx"
server_key: "d95ca1798e96a3baa3a3a26021ccf128"
server_key_iv: "1e9424a7b4d944a47c5f3a39c3e33fe8"
server_key_version: "encryptionkey@0"
I20260929 00:59:17.936740 20945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 00:59:17.940246 20960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:17.941675 20945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:17.941861 20945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "a80249ac31964c54bc77bcb6a2b120dd"
format_stamp: "Formatted at 2026-09-29 00:59:17 on dist-test-slave-pzsx"
server_key: "d95ca1798e96a3baa3a3a26021ccf128"
server_key_iv: "1e9424a7b4d944a47c5f3a39c3e33fe8"
server_key_version: "encryptionkey@0"
I20260929 00:59:17.942066 20945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:17.947881 20945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:17.951931 20945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:17.952229 20945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:17.971042 20945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44895
I20260929 00:59:17.971094 21011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44895 every 8 connection(s)
I20260929 00:59:17.972628 20945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:17.976425 21012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:17.981832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 20945
I20260929 00:59:17.982012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:17.982424 388 external_mini_cluster.cc:1428] Setting key f3768b53a4bc89908989884a0be6db02
I20260929 00:59:17.988329 21012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd: Bootstrap starting.
I20260929 00:59:17.991492 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.984301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47795 (local address 127.0.0.1:44895)
0929 00:59:17.985243 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.985266 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:17.985317 (+ 51us) server_negotiation.cc:407] Connection header received
0929 00:59:17.987670 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:17.987699 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:17.988134 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:17.988582 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:17.989579 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:17.989587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:17.989604 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:17.989622 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:17.989690 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:17.989716 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:17.989927 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:17.990071 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 00:59:17.990290 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":447,"threads_started":1}
I20260929 00:59:17.992296 21012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:17.993404 21012 log.cc:824] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:17.995342 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:17.996364 21012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd: No bootstrap required, opened a new log
I20260929 00:59:18.002104 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:17.999658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38031 (local address 127.0.0.1:44895)
0929 00:59:17.999896 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:17.999903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:18.000233 (+ 330us) server_negotiation.cc:407] Connection header received
0929 00:59:18.000383 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:18.000387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:18.000432 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:18.000575 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:18.001460 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:18.001464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:18.001467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:18.001481 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:18.001517 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:18.001532 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:18.001627 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:18.001750 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 00:59:18.001797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:59:18.005405 21012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 00:59:18.005846 21012 raft_consensus.cc:374] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:18.005955 21012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80249ac31964c54bc77bcb6a2b120dd, State: Initialized, Role: FOLLOWER
I20260929 00:59:18.006536 21012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 00:59:18.006780 21012 raft_consensus.cc:388] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:18.006934 21012 raft_consensus.cc:482] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:18.007125 21012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:18.009517 21012 raft_consensus.cc:504] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 00:59:18.009953 21012 leader_election.cc:302] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a80249ac31964c54bc77bcb6a2b120dd; no voters:
I20260929 00:59:18.010461 21012 leader_election.cc:288] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:18.010594 21022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:18.011812 21022 raft_consensus.cc:686] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [term 1 LEADER]: Becoming Leader. State: Replica: a80249ac31964c54bc77bcb6a2b120dd, State: Running, Role: LEADER
I20260929 00:59:18.012373 21022 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 00:59:18.012625 21012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:18.015784 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a80249ac31964c54bc77bcb6a2b120dd. Latest consensus state: current_term: 1 leader_uuid: "a80249ac31964c54bc77bcb6a2b120dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } } }
I20260929 00:59:18.016237 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:18.016868 21023 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a80249ac31964c54bc77bcb6a2b120dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80249ac31964c54bc77bcb6a2b120dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } } }
I20260929 00:59:18.017227 21023 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:18.020987 21029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:18.026276 21029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:18.038362 21029 catalog_manager.cc:1347] Generated new cluster ID: 6dc7033c1ba5468ebb41fe18570f65fa
I20260929 00:59:18.038836 21029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:18.082564 21029 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:18.083096 21029 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:18.095001 21029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a80249ac31964c54bc77bcb6a2b120dd: Generated new TSK 0
I20260929 00:59:18.096006 21029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:18.103344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20945
W20260929 00:59:18.104475 21007 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47795
W20260929 00:59:18.105342 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 20945: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:18.105545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 20945
2026-09-29T00:59:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:18 dist-test-slave-pzsx krb5kdc[21043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:18 dist-test-slave-pzsx krb5kdc[21043](info): set up 2 sockets
Sep 29 00:59:18 dist-test-slave-pzsx krb5kdc[21043](info): commencing operation
krb5kdc: starting...
W20260929 00:59:19.212949 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:19 dist-test-slave-pzsx krb5kdc[21043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:19.467988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:19.797612 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:19.797957 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:19.798118 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:19.805027 21056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:19.805167 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:19.805275 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:19.805387 21056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:19.805472 21056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:19.815589 21056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:19.816959 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:19.818256 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:19.829731 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.829658 21062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:19.829707 21064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:19.830842 21056 server_base.cc:1034] running on GCE node
I20260929 00:59:19.831598 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:19.833053 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:19.834282 21056 hybrid_clock.cc:648] HybridClock initialized: now 1790643559834237 us; error 37 us; skew 500 ppm
I20260929 00:59:19.834633 21056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:19 dist-test-slave-pzsx krb5kdc[21043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:19.837707 21056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:19.839217 21056 webserver.cc:460] Webserver started at http://127.0.0.1:41815/ using document root <none> and password file <none>
I20260929 00:59:19.839907 21056 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:19.840052 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:19.840358 21056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:19.843277 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "a1557f725e7e47aea1b9598246b7ae6f"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-pzsx"
server_key: "cfb2403b40070c79e593ab123a260ddf"
server_key_iv: "45ce6d457b38e209232b81f805231e93"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.844020 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "a1557f725e7e47aea1b9598246b7ae6f"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-pzsx"
server_key: "cfb2403b40070c79e593ab123a260ddf"
server_key_iv: "45ce6d457b38e209232b81f805231e93"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.848989 21056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 00:59:19.852308 21071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:19.853658 21056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:19.853855 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "a1557f725e7e47aea1b9598246b7ae6f"
format_stamp: "Formatted at 2026-09-29 00:59:19 on dist-test-slave-pzsx"
server_key: "cfb2403b40070c79e593ab123a260ddf"
server_key_iv: "45ce6d457b38e209232b81f805231e93"
server_key_version: "encryptionkey@0"
I20260929 00:59:19.854051 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:19.872946 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:19.876965 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:19.877257 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:19.895589 21056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42875
I20260929 00:59:19.895612 21122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42875 every 8 connection(s)
I20260929 00:59:19.897051 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:19.900906 21123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:19.903388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21056
I20260929 00:59:19.903580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:19.903986 388 external_mini_cluster.cc:1428] Setting key e5986a116a2d2653cfb98138100c27f5
I20260929 00:59:19.912588 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f: Bootstrap starting.
I20260929 00:59:19.916333 21123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:19.917695 21123 log.cc:824] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f: Log is configured to *not* fsync() on all Append() calls
Sep 29 00:59:19 dist-test-slave-pzsx krb5kdc[21043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:59:19.920852 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f: No bootstrap required, opened a new log
I20260929 00:59:19.927050 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.905752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37879 (local address 127.0.0.1:42875)
0929 00:59:19.906540 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.906551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.906592 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:59:19.908233 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.908255 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.908733 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.909052 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.910799 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.912089 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.913862 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.914508 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.914640 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.919733 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.919741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.919755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.919789 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.922924 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.923729 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.923738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.923744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.923939 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.924350 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.924353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.924355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.924951 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.925162 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.925615 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.926000 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":381,"threads_started":1}
I20260929 00:59:19.930215 21123 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260929 00:59:19.930645 21123 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:19.930763 21123 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1557f725e7e47aea1b9598246b7ae6f, State: Initialized, Role: FOLLOWER
I20260929 00:59:19.931064 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:19.931401 21123 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260929 00:59:19.931632 21123 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:19.931779 21123 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:19.931947 21123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:19.934335 21123 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260929 00:59:19.934821 21123 leader_election.cc:302] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1557f725e7e47aea1b9598246b7ae6f; no voters:
I20260929 00:59:19.935494 21123 leader_election.cc:288] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:19.935618 21132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:19.937570 21132 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [term 1 LEADER]: Becoming Leader. State: Replica: a1557f725e7e47aea1b9598246b7ae6f, State: Running, Role: LEADER
I20260929 00:59:19.937799 21123 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:19.938242 21132 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260929 00:59:19.947494 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1557f725e7e47aea1b9598246b7ae6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } } }
I20260929 00:59:19.947907 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:19.948385 21135 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1557f725e7e47aea1b9598246b7ae6f. Latest consensus state: current_term: 1 leader_uuid: "a1557f725e7e47aea1b9598246b7ae6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1557f725e7e47aea1b9598246b7ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } } }
I20260929 00:59:19.948907 21135 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:19.951159 21144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:19.955838 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:19.935586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54023 (local address 127.0.0.1:42875)
0929 00:59:19.935772 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:19.935781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:19.937057 (+ 1276us) server_negotiation.cc:407] Connection header received
0929 00:59:19.937262 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:19.937270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:19.937378 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:19.937505 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:19.939560 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.941107 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.943574 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:19.944570 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:19.944689 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:19.948349 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:19.948358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:19.948369 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:19.948415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:19.952663 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.953507 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.953512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.953515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.953663 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:19.954105 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:19.954110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:19.954113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:19.954600 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:19.954863 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:19.955253 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 00:59:19.955482 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:19.956638 21144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:19.965404 21144 catalog_manager.cc:1347] Generated new cluster ID: 2bebd59158ad4c6e8da8ef419dc18564
I20260929 00:59:19.965519 21144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:19.991063 21144 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:19.992376 21144 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:20.005707 21144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1557f725e7e47aea1b9598246b7ae6f: Generated new TSK 0
I20260929 00:59:20.006656 21144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:20.011909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21056
I20260929 00:59:20.013038 21118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:20.108484 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21056
2026-09-29T00:59:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:20 dist-test-slave-pzsx krb5kdc[21155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:20 dist-test-slave-pzsx krb5kdc[21155](info): set up 2 sockets
Sep 29 00:59:20 dist-test-slave-pzsx krb5kdc[21155](info): commencing operation
krb5kdc: starting...
W20260929 00:59:21.224898 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:21 dist-test-slave-pzsx krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:21.472119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:21.800962 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:21.801318 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:21.801471 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:21.808907 21168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:21.809043 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:21.809134 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:21.809249 21168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:21.809322 21168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:21.819641 21168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:21.821004 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:21.822269 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:21.834174 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:21.834111 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:21.834265 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:21.835335 21168 server_base.cc:1034] running on GCE node
I20260929 00:59:21.836125 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:21.837508 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:21.838734 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790643561838697 us; error 60 us; skew 500 ppm
I20260929 00:59:21.839075 21168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:21 dist-test-slave-pzsx krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:21.842306 21168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:21.843746 21168 webserver.cc:460] Webserver started at http://127.0.0.1:38847/ using document root <none> and password file <none>
I20260929 00:59:21.844432 21168 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:21.844610 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:21.844918 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:21.847833 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "e82affa6658844558e5b370e9a0ad206"
format_stamp: "Formatted at 2026-09-29 00:59:21 on dist-test-slave-pzsx"
server_key: "d3c3dd8c156072f59f26ea97c68ff029"
server_key_iv: "96c687e9ec6e3dcce2a53f69ffc42f5f"
server_key_version: "encryptionkey@0"
I20260929 00:59:21.848670 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "e82affa6658844558e5b370e9a0ad206"
format_stamp: "Formatted at 2026-09-29 00:59:21 on dist-test-slave-pzsx"
server_key: "d3c3dd8c156072f59f26ea97c68ff029"
server_key_iv: "96c687e9ec6e3dcce2a53f69ffc42f5f"
server_key_version: "encryptionkey@0"
I20260929 00:59:21.853785 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:59:21.857334 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:21.858686 21168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:21.858878 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "e82affa6658844558e5b370e9a0ad206"
format_stamp: "Formatted at 2026-09-29 00:59:21 on dist-test-slave-pzsx"
server_key: "d3c3dd8c156072f59f26ea97c68ff029"
server_key_iv: "96c687e9ec6e3dcce2a53f69ffc42f5f"
server_key_version: "encryptionkey@0"
I20260929 00:59:21.859102 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:21.883664 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:21.887789 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:21.888120 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:21.906389 21168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38723
I20260929 00:59:21.906419 21234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38723 every 8 connection(s)
I20260929 00:59:21.907742 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:21.911377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21168
I20260929 00:59:21.911455 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:21.911577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:21.911998 388 external_mini_cluster.cc:1428] Setting key f9e9f7a63f4a58dfb50cc0bdeca5da03
I20260929 00:59:21.926331 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:21.913724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47771 (local address 127.0.0.1:38723)
0929 00:59:21.914516 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:21.914532 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:21.914580 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:59:21.916532 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:21.916559 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:21.917083 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:21.917493 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:21.917998 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:21.919962 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:21.921314 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:21.922326 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:21.922506 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:21.923590 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:21.923599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:21.923616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:21.924372 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:21.924475 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:21.924542 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:21.924667 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:21.924823 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 00:59:21.925084 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":314,"threads_started":1}
I20260929 00:59:21.926874 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206: Bootstrap starting.
I20260929 00:59:21.929983 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:21.930300 21235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:21.931238 21235 log.cc:824] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:21.934309 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206: No bootstrap required, opened a new log
I20260929 00:59:21.940689 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:21.934249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56439 (local address 127.0.0.1:38723)
0929 00:59:21.934427 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:21.934433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:21.934878 (+ 445us) server_negotiation.cc:407] Connection header received
0929 00:59:21.935028 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:21.935034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:21.935107 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:21.935210 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:21.935692 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:21.936795 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:21.938080 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:21.938749 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:21.938867 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:21.939894 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:21.939902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:21.939907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:21.939933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:21.939981 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:21.940001 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:21.940102 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:21.940232 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 00:59:21.940325 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:21.943760 21235 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } }
I20260929 00:59:21.944139 21235 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:21.944263 21235 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82affa6658844558e5b370e9a0ad206, State: Initialized, Role: FOLLOWER
I20260929 00:59:21.944905 21235 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } }
I20260929 00:59:21.945153 21235 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:21.945299 21235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:21.945474 21235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:21.947747 21235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } }
I20260929 00:59:21.948262 21235 leader_election.cc:302] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82affa6658844558e5b370e9a0ad206; no voters:
I20260929 00:59:21.948877 21235 leader_election.cc:288] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:21.949007 21245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:21.949719 21245 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [term 1 LEADER]: Becoming Leader. State: Replica: e82affa6658844558e5b370e9a0ad206, State: Running, Role: LEADER
I20260929 00:59:21.950198 21245 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } }
I20260929 00:59:21.951042 21235 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:21.953281 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82affa6658844558e5b370e9a0ad206. Latest consensus state: current_term: 1 leader_uuid: "e82affa6658844558e5b370e9a0ad206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } } }
I20260929 00:59:21.953740 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:21.954274 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82affa6658844558e5b370e9a0ad206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82affa6658844558e5b370e9a0ad206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38723 } } }
I20260929 00:59:21.954670 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:21.955819 21253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:21.962150 21253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:21.970902 21253 catalog_manager.cc:1347] Generated new cluster ID: 216c7ebc2b564a109e6dd75b379216a1
I20260929 00:59:21.971015 21253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:21.984450 21253 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:21.986407 21253 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:21.994278 21253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82affa6658844558e5b370e9a0ad206: Generated new TSK 0
I20260929 00:59:21.995249 21253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:22.001573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21168
W20260929 00:59:22.002651 21230 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47771
W20260929 00:59:22.003315 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21168: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:22.003463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21168
2026-09-29T00:59:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:22 dist-test-slave-pzsx krb5kdc[21266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:22 dist-test-slave-pzsx krb5kdc[21266](info): set up 2 sockets
Sep 29 00:59:22 dist-test-slave-pzsx krb5kdc[21266](info): commencing operation
krb5kdc: starting...
W20260929 00:59:23.104882 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:23 dist-test-slave-pzsx krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:23.349164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:23.681386 21279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:23.681686 21279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:23.681777 21279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:23.688827 21279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:23.688928 21279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:23.688974 21279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:23.689034 21279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:23.689078 21279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:23.698606 21279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:23.699774 21279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:23.701025 21279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:23.712777 21284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:23.712814 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:23.712718 21287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:23.713804 21279 server_base.cc:1034] running on GCE node
I20260929 00:59:23.714558 21279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:23.715888 21279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:23.717078 21279 hybrid_clock.cc:648] HybridClock initialized: now 1790643563717044 us; error 24 us; skew 500 ppm
I20260929 00:59:23.717388 21279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:23 dist-test-slave-pzsx krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:23.724161 21279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:23.725757 21279 webserver.cc:460] Webserver started at http://127.0.0.1:41905/ using document root <none> and password file <none>
I20260929 00:59:23.726468 21279 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:23.726598 21279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:23.726922 21279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:23.729947 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "f7aae1ddce60434b9e20a9257d7d9bf3"
format_stamp: "Formatted at 2026-09-29 00:59:23 on dist-test-slave-pzsx"
server_key: "e4bc525d56aba5ab1feb9495a928f4ab"
server_key_iv: "0fb9438a7541ed8f8dc74fec0a0e2c7b"
server_key_version: "encryptionkey@0"
I20260929 00:59:23.730696 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "f7aae1ddce60434b9e20a9257d7d9bf3"
format_stamp: "Formatted at 2026-09-29 00:59:23 on dist-test-slave-pzsx"
server_key: "e4bc525d56aba5ab1feb9495a928f4ab"
server_key_iv: "0fb9438a7541ed8f8dc74fec0a0e2c7b"
server_key_version: "encryptionkey@0"
I20260929 00:59:23.735786 21279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:23.739120 21294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:23.740557 21279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:59:23.740746 21279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "f7aae1ddce60434b9e20a9257d7d9bf3"
format_stamp: "Formatted at 2026-09-29 00:59:23 on dist-test-slave-pzsx"
server_key: "e4bc525d56aba5ab1feb9495a928f4ab"
server_key_iv: "0fb9438a7541ed8f8dc74fec0a0e2c7b"
server_key_version: "encryptionkey@0"
I20260929 00:59:23.740934 21279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:23.746618 21279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:23.750836 21279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:23.751137 21279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:23.770063 21279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36407
I20260929 00:59:23.770113 21345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36407 every 8 connection(s)
I20260929 00:59:23.771569 21279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:23.773211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21279
I20260929 00:59:23.773380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:23.773911 388 external_mini_cluster.cc:1428] Setting key ce9678777c818f8135c1bebf8302de81
I20260929 00:59:23.776631 21346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:23.782114 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:23.776023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43847 (local address 127.0.0.1:36407)
0929 00:59:23.776986 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:23.776999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:23.777041 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:59:23.778685 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:23.778727 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:23.779074 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:23.779415 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:23.780491 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:23.780539 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:23.780563 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:23.780579 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:23.780654 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:23.780675 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:23.780852 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:23.780961 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 00:59:23.781161 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":458,"threads_started":1}
I20260929 00:59:23.785651 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:23.789690 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3: Bootstrap starting.
I20260929 00:59:23.793213 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:23.790761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59613 (local address 127.0.0.1:36407)
0929 00:59:23.790966 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:23.790971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:23.791312 (+ 341us) server_negotiation.cc:407] Connection header received
0929 00:59:23.791475 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:23.791478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:23.791524 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:23.791599 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:23.792574 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:23.792578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:23.792581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:23.792594 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:23.792629 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:23.792643 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:23.792734 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:23.792882 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 00:59:23.792925 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 00:59:23.793670 21346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:23.794713 21346 log.cc:824] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:23.797434 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3: No bootstrap required, opened a new log
I20260929 00:59:23.804426 21346 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260929 00:59:23.804895 21346 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:23.805001 21346 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7aae1ddce60434b9e20a9257d7d9bf3, State: Initialized, Role: FOLLOWER
I20260929 00:59:23.805632 21346 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260929 00:59:23.805938 21346 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:23.806110 21346 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:23.806296 21346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:23.808393 21346 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260929 00:59:23.808900 21346 leader_election.cc:302] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7aae1ddce60434b9e20a9257d7d9bf3; no voters:
I20260929 00:59:23.809438 21346 leader_election.cc:288] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:23.809593 21356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:23.810246 21356 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [term 1 LEADER]: Becoming Leader. State: Replica: f7aae1ddce60434b9e20a9257d7d9bf3, State: Running, Role: LEADER
I20260929 00:59:23.810737 21356 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } }
I20260929 00:59:23.811478 21346 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:23.813583 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7aae1ddce60434b9e20a9257d7d9bf3. Latest consensus state: current_term: 1 leader_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } } }
I20260929 00:59:23.814138 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:23.813650 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7aae1ddce60434b9e20a9257d7d9bf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36407 } } }
I20260929 00:59:23.814808 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:23.816535 21363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:23.821918 21363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:23.831414 21363 catalog_manager.cc:1347] Generated new cluster ID: 16242d746c2941059520666dbcad3a5b
I20260929 00:59:23.831554 21363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:23.863368 21363 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:23.863802 21363 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:23.873732 21363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7aae1ddce60434b9e20a9257d7d9bf3: Generated new TSK 0
I20260929 00:59:23.874747 21363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:23.889236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21279
W20260929 00:59:23.890301 21341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43847
W20260929 00:59:23.891026 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21279: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:23.891170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21279
2026-09-29T00:59:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:23 dist-test-slave-pzsx krb5kdc[21377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:23 dist-test-slave-pzsx krb5kdc[21377](info): set up 2 sockets
Sep 29 00:59:23 dist-test-slave-pzsx krb5kdc[21377](info): commencing operation
krb5kdc: starting...
W20260929 00:59:24.988876 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:25.231132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:25.551190 21390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:25.551539 21390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:25.551703 21390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:25.558753 21390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:25.558904 21390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:25.559008 21390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:25.559115 21390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:25.559199 21390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:25.568703 21390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:25.570035 21390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:25.571331 21390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:25.583006 21398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.583010 21395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:25.583495 21396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:25.583907 21390 server_base.cc:1034] running on GCE node
I20260929 00:59:25.584686 21390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:25.586028 21390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:25.587224 21390 hybrid_clock.cc:648] HybridClock initialized: now 1790643565587184 us; error 31 us; skew 500 ppm
I20260929 00:59:25.587564 21390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:25.590617 21390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:25.592139 21390 webserver.cc:460] Webserver started at http://127.0.0.1:35967/ using document root <none> and password file <none>
I20260929 00:59:25.592895 21390 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:25.593041 21390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:25.593351 21390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:25.596256 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "d08fad66e0d74d5bb2fddbba6a7010b3"
format_stamp: "Formatted at 2026-09-29 00:59:25 on dist-test-slave-pzsx"
server_key: "53333b31126c4ea3bc8d3c4e86ef62f5"
server_key_iv: "6d469832f316daa2b5735deca3406c98"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.597060 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "d08fad66e0d74d5bb2fddbba6a7010b3"
format_stamp: "Formatted at 2026-09-29 00:59:25 on dist-test-slave-pzsx"
server_key: "53333b31126c4ea3bc8d3c4e86ef62f5"
server_key_iv: "6d469832f316daa2b5735deca3406c98"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.602074 21390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:25.605489 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:25.606730 21390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:25.606915 21390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "d08fad66e0d74d5bb2fddbba6a7010b3"
format_stamp: "Formatted at 2026-09-29 00:59:25 on dist-test-slave-pzsx"
server_key: "53333b31126c4ea3bc8d3c4e86ef62f5"
server_key_iv: "6d469832f316daa2b5735deca3406c98"
server_key_version: "encryptionkey@0"
I20260929 00:59:25.607110 21390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:25.626171 21390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:25.630713 21390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:25.630998 21390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:25.649431 21390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34083
I20260929 00:59:25.649461 21456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34083 every 8 connection(s)
I20260929 00:59:25.650779 21390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:25.652603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21390
I20260929 00:59:25.652801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:25.653247 388 external_mini_cluster.cc:1428] Setting key 7919111b3846648996a71664acc548df
I20260929 00:59:25.654887 21457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:59:25.668411 21457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3: Bootstrap starting.
I20260929 00:59:25.672652 21457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:25.673913 21457 log.cc:824] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:25.677045 21457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3: No bootstrap required, opened a new log
I20260929 00:59:25.677500 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.655012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:37545 (local address 127.0.0.1:34083)
0929 00:59:25.655757 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.655772 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.655824 (+ 52us) server_negotiation.cc:407] Connection header received
0929 00:59:25.658014 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.658041 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.658561 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.659009 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.660288 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.662331 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.663594 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.664624 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.664821 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.669028 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.669038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.669056 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.669097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.673196 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.673947 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.673955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.673963 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.674161 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.674631 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.674640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.674645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.675195 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.675563 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.676039 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.676353 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":292,"threads_started":1}
I20260929 00:59:25.681212 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:25.685134 21457 raft_consensus.cc:348] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260929 00:59:25.685564 21457 raft_consensus.cc:374] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:25.685678 21457 raft_consensus.cc:729] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08fad66e0d74d5bb2fddbba6a7010b3, State: Initialized, Role: FOLLOWER
I20260929 00:59:25.686338 21457 consensus_queue.cc:260] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260929 00:59:25.686607 21457 raft_consensus.cc:388] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:25.686753 21457 raft_consensus.cc:482] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:25.686933 21457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:25.689738 21457 raft_consensus.cc:504] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260929 00:59:25.690289 21457 leader_election.cc:302] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d08fad66e0d74d5bb2fddbba6a7010b3; no voters:
I20260929 00:59:25.690936 21457 leader_election.cc:288] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:25.691058 21467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:25.692926 21457 sys_catalog.cc:564] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:25.696239 21467 raft_consensus.cc:686] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [term 1 LEADER]: Becoming Leader. State: Replica: d08fad66e0d74d5bb2fddbba6a7010b3, State: Running, Role: LEADER
I20260929 00:59:25.696959 21467 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260929 00:59:25.699836 21468 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260929 00:59:25.700253 21468 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:25.704420 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:25.685227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:45907 (local address 127.0.0.1:34083)
0929 00:59:25.685397 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:25.685403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:25.685943 (+ 540us) server_negotiation.cc:407] Connection header received
0929 00:59:25.686120 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:25.686125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:25.686201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:25.686297 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:25.687731 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.688915 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.690154 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:25.690836 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:25.690983 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:25.694868 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:25.694882 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:25.694890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:25.694945 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:25.699218 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.700323 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.700328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.700331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.700591 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:25.701323 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:25.701331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:25.701337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:25.701827 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:25.702175 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:25.703685 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0929 00:59:25.703905 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:59:25.707635 21469 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d08fad66e0d74d5bb2fddbba6a7010b3. Latest consensus state: current_term: 1 leader_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08fad66e0d74d5bb2fddbba6a7010b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260929 00:59:25.707914 21469 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:25.708596 21478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:25.713689 21478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:25.722350 21478 catalog_manager.cc:1347] Generated new cluster ID: 2943470a65c7477dbc9ed58d3100e06c
I20260929 00:59:25.722464 21478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:25.732183 21478 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:25.733441 21478 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:25.741204 21478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d08fad66e0d74d5bb2fddbba6a7010b3: Generated new TSK 0
I20260929 00:59:25.741971 21478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:25.751145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21390
I20260929 00:59:25.752931 21452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:25.841845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21390
W20260929 00:59:25.858903 21375 connection.cc:535] client connection to 127.0.0.1:34083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34083): Connection reset by peer (error 104)
2026-09-29T00:59:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21490](info): set up 2 sockets
Sep 29 00:59:25 dist-test-slave-pzsx krb5kdc[21490](info): commencing operation
krb5kdc: starting...
W20260929 00:59:26.932978 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:27 dist-test-slave-pzsx krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:27.169258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:27.491955 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:27.492295 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:27.492449 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:27.499655 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:27.499800 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:27.499904 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:27.500028 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:27.500105 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:27.509855 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:27.511073 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:27.512722 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:27.525235 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:27.525297 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:27.525321 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:27.526219 21503 server_base.cc:1034] running on GCE node
I20260929 00:59:27.526970 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:27.528200 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:27.529386 21503 hybrid_clock.cc:648] HybridClock initialized: now 1790643567529330 us; error 48 us; skew 500 ppm
I20260929 00:59:27.529769 21503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:27 dist-test-slave-pzsx krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:27.532810 21503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:27.534119 21503 webserver.cc:460] Webserver started at http://127.0.0.1:39181/ using document root <none> and password file <none>
I20260929 00:59:27.534741 21503 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:27.534868 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:27.535166 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:27.537966 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "5785725212684c7c89ffc638617fd83e"
format_stamp: "Formatted at 2026-09-29 00:59:27 on dist-test-slave-pzsx"
server_key: "15faa68cd5b2442de06ce3db519b1a34"
server_key_iv: "8b17a6e752060c11707187f587ae4d0a"
server_key_version: "encryptionkey@0"
I20260929 00:59:27.538736 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "5785725212684c7c89ffc638617fd83e"
format_stamp: "Formatted at 2026-09-29 00:59:27 on dist-test-slave-pzsx"
server_key: "15faa68cd5b2442de06ce3db519b1a34"
server_key_iv: "8b17a6e752060c11707187f587ae4d0a"
server_key_version: "encryptionkey@0"
I20260929 00:59:27.543579 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 00:59:27.546850 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:27.548054 21503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:27.548238 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "5785725212684c7c89ffc638617fd83e"
format_stamp: "Formatted at 2026-09-29 00:59:27 on dist-test-slave-pzsx"
server_key: "15faa68cd5b2442de06ce3db519b1a34"
server_key_iv: "8b17a6e752060c11707187f587ae4d0a"
server_key_version: "encryptionkey@0"
I20260929 00:59:27.548398 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:27.561668 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:27.565526 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:27.565835 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:27.583918 21503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45199
I20260929 00:59:27.583953 21569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45199 every 8 connection(s)
I20260929 00:59:27.585246 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:27.589668 21570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:27.590842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21503
I20260929 00:59:27.591037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:27.591588 388 external_mini_cluster.cc:1428] Setting key 3fd08ca6ff986e07ca46c9f17bb1301e
I20260929 00:59:27.602841 21570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e: Bootstrap starting.
I20260929 00:59:27.604964 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.593379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:60529 (local address 127.0.0.1:45199)
0929 00:59:27.594177 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:27.594188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:27.594232 (+ 44us) server_negotiation.cc:407] Connection header received
0929 00:59:27.595961 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:27.595985 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:27.596380 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:27.596871 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:27.597724 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:27.598966 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:27.600348 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:27.601152 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:27.601346 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:27.602633 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:27.602641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:27.602653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:27.603201 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:27.603276 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:27.603304 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:27.603474 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:27.603754 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 00:59:27.603960 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":377,"threads_started":1}
I20260929 00:59:27.606308 21570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:27.607590 21570 log.cc:824] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:27.609036 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:27.610865 21570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e: No bootstrap required, opened a new log
I20260929 00:59:27.620585 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:27.614156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:34491 (local address 127.0.0.1:45199)
0929 00:59:27.614349 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:27.614356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:27.614667 (+ 311us) server_negotiation.cc:407] Connection header received
0929 00:59:27.614798 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:27.614803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:27.614889 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:27.615012 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:27.615481 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:27.616680 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:27.617947 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:27.618637 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:27.618775 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:27.619826 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:27.619833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:27.619838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:27.619859 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:27.619914 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:27.619933 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:27.620053 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:27.620208 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 00:59:27.620258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:27.620932 21570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260929 00:59:27.621385 21570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:27.621500 21570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5785725212684c7c89ffc638617fd83e, State: Initialized, Role: FOLLOWER
I20260929 00:59:27.622108 21570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260929 00:59:27.622309 21570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:27.622408 21570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:27.622547 21570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:27.624756 21570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260929 00:59:27.625196 21570 leader_election.cc:302] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5785725212684c7c89ffc638617fd83e; no voters:
I20260929 00:59:27.625746 21570 leader_election.cc:288] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:27.625907 21580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:27.626579 21580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [term 1 LEADER]: Becoming Leader. State: Replica: 5785725212684c7c89ffc638617fd83e, State: Running, Role: LEADER
I20260929 00:59:27.627126 21580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260929 00:59:27.627872 21570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:27.629878 21582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5785725212684c7c89ffc638617fd83e. Latest consensus state: current_term: 1 leader_uuid: "5785725212684c7c89ffc638617fd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } } }
I20260929 00:59:27.630214 21582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:27.631106 21586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:27.630975 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5785725212684c7c89ffc638617fd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5785725212684c7c89ffc638617fd83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } } }
I20260929 00:59:27.632850 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:27.636904 21586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:27.645486 21586 catalog_manager.cc:1347] Generated new cluster ID: 58dd527abd87448586aaa450d646b5b9
I20260929 00:59:27.645613 21586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:27.662581 21586 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:27.664237 21586 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:27.676443 21586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5785725212684c7c89ffc638617fd83e: Generated new TSK 0
I20260929 00:59:27.677614 21586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:27.687541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21503
W20260929 00:59:27.688719 21565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:60529
W20260929 00:59:27.689512 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:27.689672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21503
W20260929 00:59:27.706537 21488 connection.cc:535] client connection to 127.0.0.1:45199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45199): Connection reset by peer (error 104)
2026-09-29T00:59:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:27 dist-test-slave-pzsx krb5kdc[21601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:27 dist-test-slave-pzsx krb5kdc[21601](info): set up 2 sockets
Sep 29 00:59:27 dist-test-slave-pzsx krb5kdc[21601](info): commencing operation
krb5kdc: starting...
W20260929 00:59:28.800936 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:28 dist-test-slave-pzsx krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:29.035840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:29.359601 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:29.359858 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:29.360023 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:29.366932 21614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:29.367023 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:29.367115 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:29.367192 21614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:29.367254 21614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:29.376586 21614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:29.377754 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:29.378968 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:29.390264 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:29.390336 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:29.390302 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:29.391361 21614 server_base.cc:1034] running on GCE node
I20260929 00:59:29.392166 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:29.393504 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:29.394732 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790643569394703 us; error 43 us; skew 500 ppm
I20260929 00:59:29.395066 21614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:29 dist-test-slave-pzsx krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:29.402601 21614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:29.403993 21614 webserver.cc:460] Webserver started at http://127.0.0.1:33113/ using document root <none> and password file <none>
I20260929 00:59:29.404760 21614 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:29.404903 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:29.405212 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:29.408169 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "15eb1c8460f748738c6acb1adec6bb8b"
format_stamp: "Formatted at 2026-09-29 00:59:29 on dist-test-slave-pzsx"
server_key: "0e7f5c8dec37f7ae390c05d1ca89227e"
server_key_iv: "4bd18ebcf70c65f60c18c76da84ffb35"
server_key_version: "encryptionkey@0"
I20260929 00:59:29.408946 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "15eb1c8460f748738c6acb1adec6bb8b"
format_stamp: "Formatted at 2026-09-29 00:59:29 on dist-test-slave-pzsx"
server_key: "0e7f5c8dec37f7ae390c05d1ca89227e"
server_key_iv: "4bd18ebcf70c65f60c18c76da84ffb35"
server_key_version: "encryptionkey@0"
I20260929 00:59:29.414410 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 00:59:29.417723 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:29.418990 21614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:29.419183 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "15eb1c8460f748738c6acb1adec6bb8b"
format_stamp: "Formatted at 2026-09-29 00:59:29 on dist-test-slave-pzsx"
server_key: "0e7f5c8dec37f7ae390c05d1ca89227e"
server_key_iv: "4bd18ebcf70c65f60c18c76da84ffb35"
server_key_version: "encryptionkey@0"
I20260929 00:59:29.419377 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:29.424922 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:29.429553 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:29.429860 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:29.447718 21614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35093
I20260929 00:59:29.447770 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35093 every 8 connection(s)
I20260929 00:59:29.449179 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:29.452934 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:29.459030 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21614
I20260929 00:59:29.459206 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:29.459719 388 external_mini_cluster.cc:1428] Setting key 245576a7c61ddd8413262ffbe0a30854
I20260929 00:59:29.463725 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b: Bootstrap starting.
I20260929 00:59:29.467283 21681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:29.467356 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:29.461580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:50455 (local address 127.0.0.1:35093)
0929 00:59:29.462369 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:29.462380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:29.462420 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:59:29.464131 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:29.464151 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:29.464463 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:29.464835 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:29.465871 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:29.465877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:29.465892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:29.465908 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:29.465962 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:29.465983 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:29.466139 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:29.466263 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 00:59:29.466456 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":367,"threads_started":1}
I20260929 00:59:29.468355 21681 log.cc:824] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:29.470969 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:29.471364 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b: No bootstrap required, opened a new log
I20260929 00:59:29.477763 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:29.475214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:60855 (local address 127.0.0.1:35093)
0929 00:59:29.475416 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:29.475422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:29.475742 (+ 320us) server_negotiation.cc:407] Connection header received
0929 00:59:29.475889 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:29.475892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:29.475935 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:29.476010 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:29.477086 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:29.477090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:29.477093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:29.477108 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:29.477146 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:29.477161 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:29.477269 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:29.477394 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 00:59:29.477440 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 00:59:29.479862 21681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260929 00:59:29.480227 21681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:29.480345 21681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15eb1c8460f748738c6acb1adec6bb8b, State: Initialized, Role: FOLLOWER
I20260929 00:59:29.481036 21681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260929 00:59:29.481282 21681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:29.481385 21681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:29.481503 21681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:29.483566 21681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260929 00:59:29.483989 21681 leader_election.cc:302] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15eb1c8460f748738c6acb1adec6bb8b; no voters:
I20260929 00:59:29.484596 21681 leader_election.cc:288] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:29.484745 21691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:29.485415 21691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [term 1 LEADER]: Becoming Leader. State: Replica: 15eb1c8460f748738c6acb1adec6bb8b, State: Running, Role: LEADER
I20260929 00:59:29.486173 21691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } }
I20260929 00:59:29.486857 21681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:29.489197 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15eb1c8460f748738c6acb1adec6bb8b. Latest consensus state: current_term: 1 leader_uuid: "15eb1c8460f748738c6acb1adec6bb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } } }
I20260929 00:59:29.489694 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:29.490720 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15eb1c8460f748738c6acb1adec6bb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15eb1c8460f748738c6acb1adec6bb8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35093 } } }
I20260929 00:59:29.491163 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:29.492367 21699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:29.497465 21699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:29.506192 21699 catalog_manager.cc:1347] Generated new cluster ID: 5e31fbc7a62c4f80ad74befb4aaaf759
I20260929 00:59:29.506299 21699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:29.527621 21699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:29.528117 21699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:29.549223 21699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15eb1c8460f748738c6acb1adec6bb8b: Generated new TSK 0
I20260929 00:59:29.550273 21699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:29.557016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21614
W20260929 00:59:29.558758 21676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:50455
W20260929 00:59:29.559458 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:29.559609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21614
2026-09-29T00:59:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:29 dist-test-slave-pzsx krb5kdc[21712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:29 dist-test-slave-pzsx krb5kdc[21712](info): set up 2 sockets
Sep 29 00:59:29 dist-test-slave-pzsx krb5kdc[21712](info): commencing operation
krb5kdc: starting...
W20260929 00:59:30.669051 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:30 dist-test-slave-pzsx krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:30.915460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:31.239797 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:31.240079 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:31.240170 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:31.247138 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:31.247246 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:31.247290 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:31.247349 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:31.247391 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:31.256771 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:31.257970 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:31.259173 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:31.270455 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:31.270412 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:31.270483 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:31.271360 21725 server_base.cc:1034] running on GCE node
I20260929 00:59:31.272029 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:31.273391 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:31.274600 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790643571274548 us; error 37 us; skew 500 ppm
I20260929 00:59:31.274938 21725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:31 dist-test-slave-pzsx krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:31.278234 21725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:31.279667 21725 webserver.cc:460] Webserver started at http://127.0.0.1:38361/ using document root <none> and password file <none>
I20260929 00:59:31.280344 21725 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:31.280485 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:31.280835 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:31.283676 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "5a434e3b872d4b99b6724bce8d0f065c"
format_stamp: "Formatted at 2026-09-29 00:59:31 on dist-test-slave-pzsx"
server_key: "0019f83865cd26615ef5ddb8f4521a1b"
server_key_iv: "a4e499a1b8d1f851b20ad1cb9a28d97c"
server_key_version: "encryptionkey@0"
I20260929 00:59:31.284440 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a434e3b872d4b99b6724bce8d0f065c"
format_stamp: "Formatted at 2026-09-29 00:59:31 on dist-test-slave-pzsx"
server_key: "0019f83865cd26615ef5ddb8f4521a1b"
server_key_iv: "a4e499a1b8d1f851b20ad1cb9a28d97c"
server_key_version: "encryptionkey@0"
I20260929 00:59:31.289459 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 00:59:31.292860 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:31.294238 21725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 00:59:31.294437 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "5a434e3b872d4b99b6724bce8d0f065c"
format_stamp: "Formatted at 2026-09-29 00:59:31 on dist-test-slave-pzsx"
server_key: "0019f83865cd26615ef5ddb8f4521a1b"
server_key_iv: "a4e499a1b8d1f851b20ad1cb9a28d97c"
server_key_version: "encryptionkey@0"
I20260929 00:59:31.294612 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:31.309341 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:31.313210 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:31.313432 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:31.331571 21725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35469
I20260929 00:59:31.331589 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35469 every 8 connection(s)
I20260929 00:59:31.333076 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:31.336915 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:31.337431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21725
I20260929 00:59:31.337617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:31.338048 388 external_mini_cluster.cc:1428] Setting key 2a33d2124fe70c4b74dff792de783031
I20260929 00:59:31.351523 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c: Bootstrap starting.
Sep 29 00:59:31 dist-test-slave-pzsx krb5kdc[21712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 00:59:31.354700 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:31.356068 21792 log.cc:824] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:31.359530 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c: No bootstrap required, opened a new log
I20260929 00:59:31.363024 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:31.339938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:46139 (local address 127.0.0.1:35469)
0929 00:59:31.340707 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:31.340723 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:31.340781 (+ 58us) server_negotiation.cc:407] Connection header received
0929 00:59:31.343653 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:31.343684 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:31.344234 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:31.344728 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:31.346057 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:31.348088 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:31.349411 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:31.350449 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:31.350625 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:31.354627 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:31.354638 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:31.354658 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:31.354720 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:31.358464 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:31.359196 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:31.359204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:31.359212 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:31.359416 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:31.359875 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:31.359879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:31.359882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:31.360630 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:31.360950 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:31.361314 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 00:59:31.361660 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":295,"threads_started":1}
I20260929 00:59:31.366864 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:31.369146 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260929 00:59:31.369616 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:31.369719 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a434e3b872d4b99b6724bce8d0f065c, State: Initialized, Role: FOLLOWER
I20260929 00:59:31.370409 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260929 00:59:31.370739 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:31.370903 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:31.371115 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:31.374691 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260929 00:59:31.375291 21792 leader_election.cc:302] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a434e3b872d4b99b6724bce8d0f065c; no voters:
I20260929 00:59:31.375933 21792 leader_election.cc:288] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:31.376008 21802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:31.378163 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:31.382958 21802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [term 1 LEADER]: Becoming Leader. State: Replica: 5a434e3b872d4b99b6724bce8d0f065c, State: Running, Role: LEADER
I20260929 00:59:31.383620 21802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } }
I20260929 00:59:31.386550 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a434e3b872d4b99b6724bce8d0f065c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } } }
I20260929 00:59:31.386930 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:31.388430 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a434e3b872d4b99b6724bce8d0f065c. Latest consensus state: current_term: 1 leader_uuid: "5a434e3b872d4b99b6724bce8d0f065c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a434e3b872d4b99b6724bce8d0f065c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35469 } } }
I20260929 00:59:31.388851 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:31.395015 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:31.373807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:44327 (local address 127.0.0.1:35469)
0929 00:59:31.374028 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:31.374034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:31.374368 (+ 334us) server_negotiation.cc:407] Connection header received
0929 00:59:31.374544 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:31.374548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:31.374604 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:31.374688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:31.376655 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:31.377782 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:31.379870 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:31.380897 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:31.381067 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:31.384802 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:31.384812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:31.384820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:31.384871 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:31.389797 (+ 4926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:31.391078 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:31.391086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:31.391092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:31.391263 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:31.391958 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:31.391966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:31.391972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:31.392421 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:31.394020 (+ 1599us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:31.394243 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:59:31.394477 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 00:59:31.398056 21815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:31.403262 21815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:31.412731 21815 catalog_manager.cc:1347] Generated new cluster ID: 3162ecd842704193ae3373b0ac9a13ca
I20260929 00:59:31.412868 21815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:31.431918 21815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:31.433332 21815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:31.447729 21815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a434e3b872d4b99b6724bce8d0f065c: Generated new TSK 0
I20260929 00:59:31.448664 21815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:31.463330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21725
I20260929 00:59:31.464597 21787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:31.559536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21725
W20260929 00:59:31.576882 21710 connection.cc:535] client connection to 127.0.0.1:35469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35469): Connection reset by peer (error 104)
2026-09-29T00:59:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:31 dist-test-slave-pzsx krb5kdc[21824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:31 dist-test-slave-pzsx krb5kdc[21824](info): set up 2 sockets
Sep 29 00:59:31 dist-test-slave-pzsx krb5kdc[21824](info): commencing operation
krb5kdc: starting...
W20260929 00:59:32.664985 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:32 dist-test-slave-pzsx krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:32.907923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:33.225968 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:33.226243 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:33.226334 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:33.233364 21837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:33.233476 21837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:33.233582 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:33.233693 21837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:33.233753 21837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:33.243228 21837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:33.244465 21837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:33.245811 21837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:33.257817 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:33.257836 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:33.257794 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:33.258788 21837 server_base.cc:1034] running on GCE node
I20260929 00:59:33.259475 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:33.260928 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:33.262125 21837 hybrid_clock.cc:648] HybridClock initialized: now 1790643573262079 us; error 46 us; skew 500 ppm
I20260929 00:59:33.262477 21837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:33 dist-test-slave-pzsx krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:33.265643 21837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:33.267038 21837 webserver.cc:460] Webserver started at http://127.0.0.1:35853/ using document root <none> and password file <none>
I20260929 00:59:33.267715 21837 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:33.267871 21837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:33.268172 21837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:33.271274 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "bc319a89c01a4b93971d7e04fbe5c0fa"
format_stamp: "Formatted at 2026-09-29 00:59:33 on dist-test-slave-pzsx"
server_key: "e9384e02834abc347aa36972c3cbb1e8"
server_key_iv: "ab0f0759e513955719757626e1d8c544"
server_key_version: "encryptionkey@0"
I20260929 00:59:33.272037 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc319a89c01a4b93971d7e04fbe5c0fa"
format_stamp: "Formatted at 2026-09-29 00:59:33 on dist-test-slave-pzsx"
server_key: "e9384e02834abc347aa36972c3cbb1e8"
server_key_iv: "ab0f0759e513955719757626e1d8c544"
server_key_version: "encryptionkey@0"
I20260929 00:59:33.277206 21837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 00:59:33.280591 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:33.281848 21837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:33.282027 21837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "bc319a89c01a4b93971d7e04fbe5c0fa"
format_stamp: "Formatted at 2026-09-29 00:59:33 on dist-test-slave-pzsx"
server_key: "e9384e02834abc347aa36972c3cbb1e8"
server_key_iv: "ab0f0759e513955719757626e1d8c544"
server_key_version: "encryptionkey@0"
I20260929 00:59:33.282229 21837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:33.302366 21837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:33.306716 21837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:33.307013 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:33.325120 21837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42141
I20260929 00:59:33.325155 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42141 every 8 connection(s)
I20260929 00:59:33.326464 21837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:33.329965 21904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:33.331178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21837
I20260929 00:59:33.331368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:33.331821 388 external_mini_cluster.cc:1428] Setting key c3126428a960961e50894358e9e19bc2
I20260929 00:59:33.344020 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa: Bootstrap starting.
I20260929 00:59:33.346060 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:33.333728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:42861 (local address 127.0.0.1:42141)
0929 00:59:33.334491 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:33.334506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:33.334555 (+ 49us) server_negotiation.cc:407] Connection header received
0929 00:59:33.336366 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:33.336393 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:33.336928 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:33.337349 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:33.337827 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:33.339660 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:33.341095 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:33.342137 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:33.342300 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:33.343397 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:33.343408 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:33.343424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:33.344117 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:33.344213 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:33.344242 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:33.344390 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:33.344652 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 00:59:33.344814 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":309,"threads_started":1}
I20260929 00:59:33.347564 21904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:33.348759 21904 log.cc:824] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:33.349794 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:33.351436 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa: No bootstrap required, opened a new log
I20260929 00:59:33.359346 21904 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } }
I20260929 00:59:33.359745 21904 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:33.359922 21904 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc319a89c01a4b93971d7e04fbe5c0fa, State: Initialized, Role: FOLLOWER
I20260929 00:59:33.360140 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:33.354067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:42075 (local address 127.0.0.1:42141)
0929 00:59:33.354244 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:33.354250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:33.354609 (+ 359us) server_negotiation.cc:407] Connection header received
0929 00:59:33.354821 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:33.354825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:33.354885 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:33.355001 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:33.355469 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:33.356260 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:33.357601 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:33.358060 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:33.358203 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:33.359348 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:33.359354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:33.359357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:33.359374 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:33.359409 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:33.359425 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:33.359546 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:33.359735 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 00:59:33.359796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 00:59:33.360689 21904 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } }
I20260929 00:59:33.361002 21904 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:33.361151 21904 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:33.361330 21904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:33.363574 21904 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } }
I20260929 00:59:33.364029 21904 leader_election.cc:302] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc319a89c01a4b93971d7e04fbe5c0fa; no voters:
I20260929 00:59:33.364733 21904 leader_election.cc:288] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:33.364842 21914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:33.365640 21914 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [term 1 LEADER]: Becoming Leader. State: Replica: bc319a89c01a4b93971d7e04fbe5c0fa, State: Running, Role: LEADER
I20260929 00:59:33.366160 21914 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } }
I20260929 00:59:33.366861 21904 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:33.368995 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } } }
I20260929 00:59:33.369473 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:33.369455 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc319a89c01a4b93971d7e04fbe5c0fa. Latest consensus state: current_term: 1 leader_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc319a89c01a4b93971d7e04fbe5c0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42141 } } }
I20260929 00:59:33.369817 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:33.371289 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:33.375998 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:33.384338 21921 catalog_manager.cc:1347] Generated new cluster ID: 71488509673340b6a30ff0e8f32b1f6e
I20260929 00:59:33.384444 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:33.407750 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:33.409271 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:33.417194 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc319a89c01a4b93971d7e04fbe5c0fa: Generated new TSK 0
I20260929 00:59:33.418081 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:33.425593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21837
W20260929 00:59:33.426766 21899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:42861
W20260929 00:59:33.427520 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:33.427656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21837
W20260929 00:59:33.445137 21822 connection.cc:535] client connection to 127.0.0.1:42141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42141): Connection reset by peer (error 104)
2026-09-29T00:59:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:33 dist-test-slave-pzsx krb5kdc[21935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:33 dist-test-slave-pzsx krb5kdc[21935](info): set up 2 sockets
Sep 29 00:59:33 dist-test-slave-pzsx krb5kdc[21935](info): commencing operation
krb5kdc: starting...
W20260929 00:59:34.512805 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:34 dist-test-slave-pzsx krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 00:59:34.758090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:35.083971 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:35.084296 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:35.084474 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:35.091663 21948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:35.091812 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:35.091913 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:35.092020 21948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:35.092108 21948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:35.101977 21948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:35.103238 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:35.104605 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:35.115909 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:35.115998 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:35.116186 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:35.117085 21948 server_base.cc:1034] running on GCE node
I20260929 00:59:35.117815 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:35.119189 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:35.120414 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790643575120376 us; error 57 us; skew 500 ppm
I20260929 00:59:35.120839 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:35 dist-test-slave-pzsx krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:35.134258 21948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 00:59:35.135738 21948 webserver.cc:460] Webserver started at http://127.0.0.1:38819/ using document root <none> and password file <none>
I20260929 00:59:35.136454 21948 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:35.136617 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:35.136927 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:35.139926 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "6e02b167e6544538810556059561127b"
format_stamp: "Formatted at 2026-09-29 00:59:35 on dist-test-slave-pzsx"
server_key: "e0e52ae8c66f00aeb1c290297c68d822"
server_key_iv: "3404571b4115b76d95ffb7142dfdad64"
server_key_version: "encryptionkey@0"
I20260929 00:59:35.140780 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e02b167e6544538810556059561127b"
format_stamp: "Formatted at 2026-09-29 00:59:35 on dist-test-slave-pzsx"
server_key: "e0e52ae8c66f00aeb1c290297c68d822"
server_key_iv: "3404571b4115b76d95ffb7142dfdad64"
server_key_version: "encryptionkey@0"
I20260929 00:59:35.145859 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 00:59:35.149194 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:35.150537 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 00:59:35.150720 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "6e02b167e6544538810556059561127b"
format_stamp: "Formatted at 2026-09-29 00:59:35 on dist-test-slave-pzsx"
server_key: "e0e52ae8c66f00aeb1c290297c68d822"
server_key_iv: "3404571b4115b76d95ffb7142dfdad64"
server_key_version: "encryptionkey@0"
I20260929 00:59:35.150925 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:35.156620 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:35.160990 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:35.161304 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:35.179550 21948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34591
I20260929 00:59:35.179595 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34591 every 8 connection(s)
I20260929 00:59:35.181003 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:35.182047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 21948
I20260929 00:59:35.182235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:35.182687 388 external_mini_cluster.cc:1428] Setting key cacf00c2ec452a849be8ba035642f208
I20260929 00:59:35.185367 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:35.190560 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:35.184433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:40283 (local address 127.0.0.1:34591)
0929 00:59:35.185165 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:35.185187 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:35.185227 (+ 40us) server_negotiation.cc:407] Connection header received
0929 00:59:35.186858 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:35.186878 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:35.187186 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:35.187610 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:35.188641 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:35.188647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:35.188659 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:35.188673 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:35.188738 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:35.188756 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:35.189000 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:35.189136 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 00:59:35.189314 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":302,"threads_started":1}
I20260929 00:59:35.193994 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:35.197090 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b: Bootstrap starting.
I20260929 00:59:35.200729 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:35.198231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:58607 (local address 127.0.0.1:34591)
0929 00:59:35.198377 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:35.198381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:35.198894 (+ 513us) server_negotiation.cc:407] Connection header received
0929 00:59:35.199074 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:35.199079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:35.199123 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:35.199230 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:35.200085 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:35.200096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:35.200100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:35.200118 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:35.200153 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:35.200167 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:35.200280 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:35.200393 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 00:59:35.200449 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:59:35.200737 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:35.201972 22015 log.cc:824] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:35.204461 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b: No bootstrap required, opened a new log
I20260929 00:59:35.212167 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } }
I20260929 00:59:35.212555 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:35.212666 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e02b167e6544538810556059561127b, State: Initialized, Role: FOLLOWER
I20260929 00:59:35.213193 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } }
I20260929 00:59:35.213438 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:35.213618 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:35.213781 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:35.216142 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } }
I20260929 00:59:35.216732 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e02b167e6544538810556059561127b; no voters:
I20260929 00:59:35.217275 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:35.217373 22025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:35.218067 22025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [term 1 LEADER]: Becoming Leader. State: Replica: 6e02b167e6544538810556059561127b, State: Running, Role: LEADER
I20260929 00:59:35.218650 22025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } }
I20260929 00:59:35.219360 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:35.221612 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e02b167e6544538810556059561127b. Latest consensus state: current_term: 1 leader_uuid: "6e02b167e6544538810556059561127b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } } }
I20260929 00:59:35.221778 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e02b167e6544538810556059561127b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e02b167e6544538810556059561127b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34591 } } }
I20260929 00:59:35.222098 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:35.222095 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:35.223948 22034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:35.229001 22034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:35.237303 22034 catalog_manager.cc:1347] Generated new cluster ID: 8ee68b36bf904138943c171597cc6987
I20260929 00:59:35.237413 22034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:35.259299 22034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:35.259760 22034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:35.267680 22034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e02b167e6544538810556059561127b: Generated new TSK 0
I20260929 00:59:35.268719 22034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:35.274441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21948
W20260929 00:59:35.275519 22010 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:40283
W20260929 00:59:35.276208 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 21948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:35.276331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 21948
2026-09-29T00:59:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:35 dist-test-slave-pzsx krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:35 dist-test-slave-pzsx krb5kdc[22046](info): set up 2 sockets
Sep 29 00:59:35 dist-test-slave-pzsx krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260929 00:59:36.364974 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:36 dist-test-slave-pzsx krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:36.603873 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:36.925832 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:36.926115 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:36.926311 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:36.933916 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:36.934034 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:36.934150 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:36.934262 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:36.934341 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:36.944168 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:36.945509 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:36.946902 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:36.958580 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:36.958554 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:36.958640 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:36.959654 22059 server_base.cc:1034] running on GCE node
I20260929 00:59:36.960301 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:36.961638 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:36.962849 22059 hybrid_clock.cc:648] HybridClock initialized: now 1790643576962790 us; error 47 us; skew 500 ppm
I20260929 00:59:36.963187 22059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:36 dist-test-slave-pzsx krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:36.966352 22059 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:36.967758 22059 webserver.cc:460] Webserver started at http://127.0.97.62:33221/ using document root <none> and password file <none>
I20260929 00:59:36.968406 22059 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:36.968583 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:36.968904 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:36.971791 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "e817ed5badb840768f31057811616356"
format_stamp: "Formatted at 2026-09-29 00:59:36 on dist-test-slave-pzsx"
server_key: "f644cebbdce463e017be5d773c9746d4"
server_key_iv: "2880a1a08f7a4a0cf6731798a524084c"
server_key_version: "encryptionkey@0"
I20260929 00:59:36.972608 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "e817ed5badb840768f31057811616356"
format_stamp: "Formatted at 2026-09-29 00:59:36 on dist-test-slave-pzsx"
server_key: "f644cebbdce463e017be5d773c9746d4"
server_key_iv: "2880a1a08f7a4a0cf6731798a524084c"
server_key_version: "encryptionkey@0"
I20260929 00:59:36.977636 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 00:59:36.981108 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:36.982394 22059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 00:59:36.982595 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "e817ed5badb840768f31057811616356"
format_stamp: "Formatted at 2026-09-29 00:59:36 on dist-test-slave-pzsx"
server_key: "f644cebbdce463e017be5d773c9746d4"
server_key_iv: "2880a1a08f7a4a0cf6731798a524084c"
server_key_version: "encryptionkey@0"
I20260929 00:59:36.982796 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:37.018499 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:37.022382 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:37.022671 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:37.041451 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45257
I20260929 00:59:37.041491 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45257 every 8 connection(s)
I20260929 00:59:37.042825 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:37.046486 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:37.047613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22059
I20260929 00:59:37.047803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:37.048354 388 external_mini_cluster.cc:1428] Setting key dc6ee491f6ce49ca3d94775d16bd6cfe
I20260929 00:59:37.060395 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: Bootstrap starting.
Sep 29 00:59:37 dist-test-slave-pzsx krb5kdc[22046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:37.063947 22126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:37.065176 22126 log.cc:824] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:37.067939 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: No bootstrap required, opened a new log
I20260929 00:59:37.070885 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:37.050180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57233 (local address 127.0.97.62:45257)
0929 00:59:37.050932 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:37.050943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:37.050988 (+ 45us) server_negotiation.cc:407] Connection header received
0929 00:59:37.052631 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:37.052656 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:37.053037 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:37.053364 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:37.055127 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:37.056330 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:37.058278 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:37.058914 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:37.059030 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:37.063724 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:37.063732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:37.063746 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:37.063780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:37.066856 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:37.067589 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:37.067596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:37.067601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:37.067755 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:37.068255 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:37.068265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:37.068267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:37.068796 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:37.069123 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:37.069623 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 00:59:37.069904 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":360,"threads_started":1}
I20260929 00:59:37.074932 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:37.076363 22126 raft_consensus.cc:348] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } }
I20260929 00:59:37.076776 22126 raft_consensus.cc:374] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:37.076879 22126 raft_consensus.cc:729] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e817ed5badb840768f31057811616356, State: Initialized, Role: FOLLOWER
I20260929 00:59:37.077420 22126 consensus_queue.cc:260] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } }
I20260929 00:59:37.077667 22126 raft_consensus.cc:388] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:37.077817 22126 raft_consensus.cc:482] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:37.078024 22126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:37.080690 22126 raft_consensus.cc:504] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } }
I20260929 00:59:37.081179 22126 leader_election.cc:302] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e817ed5badb840768f31057811616356; no voters:
I20260929 00:59:37.082074 22126 leader_election.cc:288] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:37.082170 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:37.084705 22126 sys_catalog.cc:564] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:37.084786 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [term 1 LEADER]: Becoming Leader. State: Replica: e817ed5badb840768f31057811616356, State: Running, Role: LEADER
I20260929 00:59:37.085422 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } }
I20260929 00:59:37.089268 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e817ed5badb840768f31057811616356. Latest consensus state: current_term: 1 leader_uuid: "e817ed5badb840768f31057811616356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } } }
I20260929 00:59:37.089807 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:37.089340 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e817ed5badb840768f31057811616356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e817ed5badb840768f31057811616356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45257 } } }
I20260929 00:59:37.090693 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:37.098687 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:37.079372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43089 (local address 127.0.97.62:45257)
0929 00:59:37.079537 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:37.079541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:37.079967 (+ 426us) server_negotiation.cc:407] Connection header received
0929 00:59:37.080166 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:37.080184 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:37.080240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:37.080337 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:37.081917 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:37.083139 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:37.084339 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:37.085053 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:37.085207 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:37.088721 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:37.088730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:37.088735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:37.088783 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:37.093062 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:37.093977 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:37.093987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:37.093990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:37.095471 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:37.096193 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:37.096201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:37.096206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:37.096739 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:37.098127 (+ 1388us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:37.098298 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 00:59:37.098457 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 00:59:37.102262 22152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:37.102386 22152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:37.102530 22150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:37.104796 22150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:37.114115 22150 catalog_manager.cc:1347] Generated new cluster ID: 14c0f673d94749babd9cd2a5bed60c2b
I20260929 00:59:37.114218 22150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:37.128007 22150 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:37.129747 22150 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:37.140146 22150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e817ed5badb840768f31057811616356: Generated new TSK 0
I20260929 00:59:37.141146 22150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:37.146708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22059
I20260929 00:59:37.148486 22121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:37.239030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22059
W20260929 00:59:37.253261 22044 connection.cc:535] client connection to 127.0.97.62:45257 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45257): Connection reset by peer (error 104)
2026-09-29T00:59:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:37 dist-test-slave-pzsx krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:37 dist-test-slave-pzsx krb5kdc[22158](info): set up 2 sockets
Sep 29 00:59:37 dist-test-slave-pzsx krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260929 00:59:38.321049 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:38 dist-test-slave-pzsx krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:38.581909 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45699
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:38.913568 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:38.913888 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:38.914070 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:38.921213 22171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:38.921316 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:38.921456 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:38.921566 22171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:38.921659 22171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:38.931569 22171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:38.932927 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:38.934305 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:38.946817 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:38.946777 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:38.947352 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:38.947785 22171 server_base.cc:1034] running on GCE node
I20260929 00:59:38.948617 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:38.949991 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:38.951232 22171 hybrid_clock.cc:648] HybridClock initialized: now 1790643578951203 us; error 50 us; skew 500 ppm
I20260929 00:59:38.951591 22171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:38 dist-test-slave-pzsx krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:38.954644 22171 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:38.956210 22171 webserver.cc:460] Webserver started at http://127.0.97.62:34223/ using document root <none> and password file <none>
I20260929 00:59:38.956949 22171 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:38.957103 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:38.957410 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:38.960992 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8"
format_stamp: "Formatted at 2026-09-29 00:59:38 on dist-test-slave-pzsx"
server_key: "7e9e5a3d27fc21d58b00b317693c1580"
server_key_iv: "004c9dcd4892c0ecf69ebc0c2be2b277"
server_key_version: "encryptionkey@0"
I20260929 00:59:38.961781 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8"
format_stamp: "Formatted at 2026-09-29 00:59:38 on dist-test-slave-pzsx"
server_key: "7e9e5a3d27fc21d58b00b317693c1580"
server_key_iv: "004c9dcd4892c0ecf69ebc0c2be2b277"
server_key_version: "encryptionkey@0"
I20260929 00:59:38.966840 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:38.970252 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:38.971487 22171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:59:38.971673 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8"
format_stamp: "Formatted at 2026-09-29 00:59:38 on dist-test-slave-pzsx"
server_key: "7e9e5a3d27fc21d58b00b317693c1580"
server_key_iv: "004c9dcd4892c0ecf69ebc0c2be2b277"
server_key_version: "encryptionkey@0"
I20260929 00:59:38.971872 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:38.998282 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:39.002256 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:39.002566 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:39.022068 22171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45699
I20260929 00:59:39.022101 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45699 every 8 connection(s)
I20260929 00:59:39.023495 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:39.026892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22171
I20260929 00:59:39.027073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:39.027379 22238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:39.027573 388 external_mini_cluster.cc:1428] Setting key 54b470170dd60bffa12a993d43163faa
I20260929 00:59:39.042743 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8: Bootstrap starting.
I20260929 00:59:39.042814 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:39.029309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35555 (local address 127.0.97.62:45699)
0929 00:59:39.030071 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:39.030083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:39.030125 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:59:39.032095 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:39.032116 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:39.032573 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:39.033038 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:39.033583 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:39.034914 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:39.036787 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:39.037448 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:39.037585 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:39.039029 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:39.039040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:39.039061 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:39.039992 (+ 931us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:39.040071 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:39.040113 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:39.040316 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:39.041113 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 00:59:39.041313 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":346,"threads_started":1}
I20260929 00:59:39.047055 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:39.047492 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:39.048833 22238 log.cc:824] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:39.052011 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8: No bootstrap required, opened a new log
I20260929 00:59:39.058915 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:39.052052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54447 (local address 127.0.97.62:45699)
0929 00:59:39.052289 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:39.052298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:39.052831 (+ 533us) server_negotiation.cc:407] Connection header received
0929 00:59:39.052975 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:39.052980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:39.053061 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:39.053170 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:39.053651 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:39.054849 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:39.056181 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:39.056884 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:39.057024 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:39.058095 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:39.058102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:39.058106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:39.058129 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:39.058198 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:39.058218 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:39.058344 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:39.058504 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 00:59:39.058563 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 00:59:39.061720 22238 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } }
I20260929 00:59:39.062100 22238 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:39.062177 22238 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc35d1c1c1a64bbe99caa7523c14d7e8, State: Initialized, Role: FOLLOWER
I20260929 00:59:39.062803 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } }
I20260929 00:59:39.063014 22238 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:39.063117 22238 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:39.063235 22238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:39.065507 22238 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } }
I20260929 00:59:39.066039 22238 leader_election.cc:302] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc35d1c1c1a64bbe99caa7523c14d7e8; no voters:
I20260929 00:59:39.066625 22238 leader_election.cc:288] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:39.066764 22248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:39.067456 22248 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [term 1 LEADER]: Becoming Leader. State: Replica: dc35d1c1c1a64bbe99caa7523c14d7e8, State: Running, Role: LEADER
I20260929 00:59:39.068017 22248 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } }
I20260929 00:59:39.069011 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:39.071210 22250 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc35d1c1c1a64bbe99caa7523c14d7e8. Latest consensus state: current_term: 1 leader_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } } }
I20260929 00:59:39.071189 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc35d1c1c1a64bbe99caa7523c14d7e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45699 } } }
I20260929 00:59:39.071650 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:39.073335 22254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:39.073387 22250 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:39.078573 22254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:39.088929 22254 catalog_manager.cc:1347] Generated new cluster ID: 1dbe3d11ac6c479d822e9cd8283d3e2b
I20260929 00:59:39.089064 22254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:39.103096 22254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:39.104364 22254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:39.115401 22254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc35d1c1c1a64bbe99caa7523c14d7e8: Generated new TSK 0
I20260929 00:59:39.116231 22254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:39.124351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22171
W20260929 00:59:39.126058 22233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35555
W20260929 00:59:39.126864 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22171: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:39.127013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22171
W20260929 00:59:39.144296 22156 connection.cc:535] client connection to 127.0.97.62:45699 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45699): Connection reset by peer (error 104)
2026-09-29T00:59:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:39 dist-test-slave-pzsx krb5kdc[22269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:39 dist-test-slave-pzsx krb5kdc[22269](info): set up 2 sockets
Sep 29 00:59:39 dist-test-slave-pzsx krb5kdc[22269](info): commencing operation
krb5kdc: starting...
W20260929 00:59:40.224962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:40 dist-test-slave-pzsx krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:40.472388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:40.812431 22282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:40.812737 22282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:40.812829 22282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:40.819931 22282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:40.820034 22282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:40.820078 22282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:40.820130 22282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:40.820166 22282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:40.829612 22282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:40.830801 22282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:40.832043 22282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:40.844126 22287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:40.844241 22290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:40.844261 22288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:40.844887 22282 server_base.cc:1034] running on GCE node
I20260929 00:59:40.845686 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:40.847100 22282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:40.848361 22282 hybrid_clock.cc:648] HybridClock initialized: now 1790643580848328 us; error 53 us; skew 500 ppm
I20260929 00:59:40.848778 22282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:40 dist-test-slave-pzsx krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:40.851754 22282 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:40.853243 22282 webserver.cc:460] Webserver started at http://127.0.97.62:41155/ using document root <none> and password file <none>
I20260929 00:59:40.853941 22282 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:40.854110 22282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:40.854426 22282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:40.857473 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "644aa2b20e644f96b4c844f7a43c421f"
format_stamp: "Formatted at 2026-09-29 00:59:40 on dist-test-slave-pzsx"
server_key: "9668624e5735e047e460587683268150"
server_key_iv: "0ffcf5858a2b2ad40889c40447610d65"
server_key_version: "encryptionkey@0"
I20260929 00:59:40.858273 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "644aa2b20e644f96b4c844f7a43c421f"
format_stamp: "Formatted at 2026-09-29 00:59:40 on dist-test-slave-pzsx"
server_key: "9668624e5735e047e460587683268150"
server_key_iv: "0ffcf5858a2b2ad40889c40447610d65"
server_key_version: "encryptionkey@0"
I20260929 00:59:40.863440 22282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:40.866878 22297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:40.868132 22282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 00:59:40.868314 22282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "644aa2b20e644f96b4c844f7a43c421f"
format_stamp: "Formatted at 2026-09-29 00:59:40 on dist-test-slave-pzsx"
server_key: "9668624e5735e047e460587683268150"
server_key_iv: "0ffcf5858a2b2ad40889c40447610d65"
server_key_version: "encryptionkey@0"
I20260929 00:59:40.868538 22282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:40.874279 22282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:40.878562 22282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:40.878870 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:40.897185 22282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45731
I20260929 00:59:40.897236 22348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45731 every 8 connection(s)
I20260929 00:59:40.898592 22282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:40.902276 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:40.906702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22282
I20260929 00:59:40.906883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:40.907287 388 external_mini_cluster.cc:1428] Setting key bc4248647d1fca6dce4a725ca90cab7a
I20260929 00:59:40.913957 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f: Bootstrap starting.
I20260929 00:59:40.916321 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:40.909115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56147 (local address 127.0.97.62:45731)
0929 00:59:40.909951 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:40.909966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:40.910014 (+ 48us) server_negotiation.cc:407] Connection header received
0929 00:59:40.912437 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:40.912463 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:40.912903 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:40.913491 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:40.914435 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:40.914448 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:40.914469 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:40.914486 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:40.914547 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:40.914571 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:40.914779 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:40.914899 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 00:59:40.915066 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":340,"threads_started":1}
I20260929 00:59:40.917536 22349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:40.918663 22349 log.cc:824] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:40.920001 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:40.921516 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f: No bootstrap required, opened a new log
I20260929 00:59:40.926798 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:40.924419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47337 (local address 127.0.97.62:45731)
0929 00:59:40.924615 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:40.924621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:40.924936 (+ 315us) server_negotiation.cc:407] Connection header received
0929 00:59:40.925091 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:40.925095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:40.925137 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:40.925216 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:40.926144 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:40.926148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:40.926151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:40.926165 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:40.926205 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:40.926220 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:40.926307 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:40.926448 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 00:59:40.926490 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 00:59:40.929924 22349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260929 00:59:40.930326 22349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:40.930444 22349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644aa2b20e644f96b4c844f7a43c421f, State: Initialized, Role: FOLLOWER
I20260929 00:59:40.931015 22349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260929 00:59:40.931278 22349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:40.931427 22349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:40.931602 22349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:40.933811 22349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260929 00:59:40.934307 22349 leader_election.cc:302] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 644aa2b20e644f96b4c844f7a43c421f; no voters:
I20260929 00:59:40.934948 22349 leader_election.cc:288] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:40.935046 22359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:40.935797 22359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [term 1 LEADER]: Becoming Leader. State: Replica: 644aa2b20e644f96b4c844f7a43c421f, State: Running, Role: LEADER
I20260929 00:59:40.936388 22359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260929 00:59:40.937089 22349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:40.939263 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "644aa2b20e644f96b4c844f7a43c421f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } } }
I20260929 00:59:40.939347 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 644aa2b20e644f96b4c844f7a43c421f. Latest consensus state: current_term: 1 leader_uuid: "644aa2b20e644f96b4c844f7a43c421f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644aa2b20e644f96b4c844f7a43c421f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } } }
I20260929 00:59:40.939662 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:40.939728 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:40.940675 22365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:40.946623 22365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:40.954715 22365 catalog_manager.cc:1347] Generated new cluster ID: 1fd68c39de084573826952effc2158d2
I20260929 00:59:40.954840 22365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:40.979066 22365 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:40.979576 22365 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:40.992095 22365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 644aa2b20e644f96b4c844f7a43c421f: Generated new TSK 0
I20260929 00:59:40.992918 22365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:41.003014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22282
W20260929 00:59:41.004624 22344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56147
W20260929 00:59:41.005359 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:41.005497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22282
2026-09-29T00:59:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:41 dist-test-slave-pzsx krb5kdc[22380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:41 dist-test-slave-pzsx krb5kdc[22380](info): set up 2 sockets
Sep 29 00:59:41 dist-test-slave-pzsx krb5kdc[22380](info): commencing operation
krb5kdc: starting...
W20260929 00:59:42.109042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:42 dist-test-slave-pzsx krb5kdc[22380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:42.359616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:42.688843 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:42.689128 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:42.689297 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:42.696370 22393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:42.696465 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:42.696599 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:42.696722 22393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:42.696808 22393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:42.706364 22393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:42.707638 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:42.708937 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:42.720604 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:42.720707 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:42.720870 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:42.721659 22393 server_base.cc:1034] running on GCE node
I20260929 00:59:42.722409 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:42.723754 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:42.724999 22393 hybrid_clock.cc:648] HybridClock initialized: now 1790643582724942 us; error 47 us; skew 500 ppm
I20260929 00:59:42.725327 22393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:42 dist-test-slave-pzsx krb5kdc[22380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:42.728619 22393 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:42.730221 22393 webserver.cc:460] Webserver started at http://127.0.97.62:43701/ using document root <none> and password file <none>
I20260929 00:59:42.730895 22393 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:42.731048 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:42.731359 22393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:42.734346 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "99f43c34871e4ef69c51dfb521b2e5e2"
format_stamp: "Formatted at 2026-09-29 00:59:42 on dist-test-slave-pzsx"
server_key: "4c6ee838e6f5091d6f4ee9e61cc42c05"
server_key_iv: "11ec680656e0b3a5b1875af8d5bdf8e4"
server_key_version: "encryptionkey@0"
I20260929 00:59:42.735095 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "99f43c34871e4ef69c51dfb521b2e5e2"
format_stamp: "Formatted at 2026-09-29 00:59:42 on dist-test-slave-pzsx"
server_key: "4c6ee838e6f5091d6f4ee9e61cc42c05"
server_key_iv: "11ec680656e0b3a5b1875af8d5bdf8e4"
server_key_version: "encryptionkey@0"
I20260929 00:59:42.740299 22393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 00:59:42.743804 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:42.745193 22393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:42.745398 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "99f43c34871e4ef69c51dfb521b2e5e2"
format_stamp: "Formatted at 2026-09-29 00:59:42 on dist-test-slave-pzsx"
server_key: "4c6ee838e6f5091d6f4ee9e61cc42c05"
server_key_iv: "11ec680656e0b3a5b1875af8d5bdf8e4"
server_key_version: "encryptionkey@0"
I20260929 00:59:42.745625 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:42.766139 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:42.770133 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:42.770444 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:42.789275 22393 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41035
I20260929 00:59:42.789318 22459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41035 every 8 connection(s)
I20260929 00:59:42.790660 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:42.794302 22460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:42.794544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22393
I20260929 00:59:42.794735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:42.795302 388 external_mini_cluster.cc:1428] Setting key 6644c212ccdf23374564c3cc36ee062f
I20260929 00:59:42.807804 22460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2: Bootstrap starting.
Sep 29 00:59:42 dist-test-slave-pzsx krb5kdc[22380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:42.811324 22460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:42.812381 22460 log.cc:824] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:42.815116 22460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2: No bootstrap required, opened a new log
I20260929 00:59:42.818620 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:42.797077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51171 (local address 127.0.97.62:41035)
0929 00:59:42.797945 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:42.797956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:42.797994 (+ 38us) server_negotiation.cc:407] Connection header received
0929 00:59:42.800264 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:42.800295 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:42.800755 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:42.801086 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:42.802760 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:42.803988 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:42.805829 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:42.806464 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:42.806580 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:42.811217 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:42.811225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:42.811241 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:42.811285 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:42.814593 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:42.815305 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:42.815311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:42.815317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:42.815481 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:42.815909 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:42.815919 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:42.815921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:42.816419 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:42.816750 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:42.817174 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 00:59:42.817534 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":421,"threads_started":1}
I20260929 00:59:42.822570 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:42.824122 22460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 00:59:42.824597 22460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:42.824716 22460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f43c34871e4ef69c51dfb521b2e5e2, State: Initialized, Role: FOLLOWER
I20260929 00:59:42.825321 22460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 00:59:42.825598 22460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:42.825743 22460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:42.825937 22460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:42.828624 22460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 00:59:42.829103 22460 leader_election.cc:302] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f43c34871e4ef69c51dfb521b2e5e2; no voters:
I20260929 00:59:42.829823 22460 leader_election.cc:288] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:42.829938 22470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:42.832332 22460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:42.832811 22470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [term 1 LEADER]: Becoming Leader. State: Replica: 99f43c34871e4ef69c51dfb521b2e5e2, State: Running, Role: LEADER
I20260929 00:59:42.833438 22470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 00:59:42.837360 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f43c34871e4ef69c51dfb521b2e5e2. Latest consensus state: current_term: 1 leader_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260929 00:59:42.837754 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:42.838346 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f43c34871e4ef69c51dfb521b2e5e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260929 00:59:42.838704 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:42.845561 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:42.826887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45867 (local address 127.0.97.62:41035)
0929 00:59:42.827063 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:42.827070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:42.827585 (+ 515us) server_negotiation.cc:407] Connection header received
0929 00:59:42.827690 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:42.827696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:42.827777 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:42.827877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:42.829328 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:42.830335 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:42.831826 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:42.832289 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:42.832443 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:42.835567 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:42.835572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:42.835576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:42.835604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:42.841669 (+ 6065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:42.842743 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:42.842751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:42.842757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:42.842942 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:42.843468 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:42.843476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:42.843482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:42.844026 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:42.844345 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:42.844745 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 00:59:42.845046 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 00:59:42.846967 22480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:42.851850 22480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:42.860884 22480 catalog_manager.cc:1347] Generated new cluster ID: 5818b476034345a4816418827499fbea
I20260929 00:59:42.860993 22480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:42.868321 22480 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:42.870132 22480 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:42.880491 22480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f43c34871e4ef69c51dfb521b2e5e2: Generated new TSK 0
I20260929 00:59:42.881454 22480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:42.889338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22393
I20260929 00:59:42.890545 22455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:42.978813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22393
2026-09-29T00:59:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:43 dist-test-slave-pzsx krb5kdc[22492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:43 dist-test-slave-pzsx krb5kdc[22492](info): set up 2 sockets
Sep 29 00:59:43 dist-test-slave-pzsx krb5kdc[22492](info): commencing operation
krb5kdc: starting...
W20260929 00:59:44.072991 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:44 dist-test-slave-pzsx krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:44.331992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:44.658865 22505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:44.659219 22505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:44.659415 22505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:44.666539 22505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:44.666695 22505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:44.666810 22505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:44.666921 22505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:44.667008 22505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:44.676637 22505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:44.677911 22505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:44.679208 22505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:44.691088 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:44.691139 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:44.691162 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:44.691929 22505 server_base.cc:1034] running on GCE node
I20260929 00:59:44.692836 22505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:44.694250 22505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:44.695489 22505 hybrid_clock.cc:648] HybridClock initialized: now 1790643584695448 us; error 38 us; skew 500 ppm
I20260929 00:59:44.695861 22505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:44 dist-test-slave-pzsx krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:44.699043 22505 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:44.700637 22505 webserver.cc:460] Webserver started at http://127.0.97.62:38887/ using document root <none> and password file <none>
I20260929 00:59:44.701325 22505 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:44.701475 22505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:44.701776 22505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:44.704766 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-pzsx"
server_key: "c3bcb1cb1a16a2972aecce8a1a0bf824"
server_key_iv: "c3cb97fc8fc07d0c7eb5e33eb594b15f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.705577 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-pzsx"
server_key: "c3bcb1cb1a16a2972aecce8a1a0bf824"
server_key_iv: "c3cb97fc8fc07d0c7eb5e33eb594b15f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.710734 22505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 00:59:44.714201 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:44.715430 22505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 00:59:44.715615 22505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf"
format_stamp: "Formatted at 2026-09-29 00:59:44 on dist-test-slave-pzsx"
server_key: "c3bcb1cb1a16a2972aecce8a1a0bf824"
server_key_iv: "c3cb97fc8fc07d0c7eb5e33eb594b15f"
server_key_version: "encryptionkey@0"
I20260929 00:59:44.715812 22505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:44.735085 22505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:44.739425 22505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:44.739727 22505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:44.758113 22505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34799
I20260929 00:59:44.758149 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34799 every 8 connection(s)
I20260929 00:59:44.759462 22505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:44.763134 22572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:44.767182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22505
I20260929 00:59:44.767377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:44.767889 388 external_mini_cluster.cc:1428] Setting key e9969be1303c88bd00c6e4a03021d20e
I20260929 00:59:44.775954 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf: Bootstrap starting.
I20260929 00:59:44.779683 22572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:44.780865 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.769686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50953 (local address 127.0.97.62:34799)
0929 00:59:44.770421 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.770432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.770473 (+ 41us) server_negotiation.cc:407] Connection header received
0929 00:59:44.772030 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.772051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.772471 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.772879 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:44.773422 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.774694 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.776576 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.777216 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.777342 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.778617 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:44.778624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:44.778638 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:44.779138 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:44.779205 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:44.779228 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:44.779308 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.779552 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.779831 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":350,"threads_started":1}
I20260929 00:59:44.780869 22572 log.cc:824] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:44.784013 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf: No bootstrap required, opened a new log
I20260929 00:59:44.784543 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:44.792689 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260929 00:59:44.793155 22572 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:44.793295 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1cd27ec511a49dab54cc2cd1f9ff3cf, State: Initialized, Role: FOLLOWER
I20260929 00:59:44.793877 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260929 00:59:44.794130 22572 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:44.794287 22572 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:44.794485 22572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:44.795223 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:44.788657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37769 (local address 127.0.97.62:34799)
0929 00:59:44.788845 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:44.788851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:44.789246 (+ 395us) server_negotiation.cc:407] Connection header received
0929 00:59:44.789406 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:44.789412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:44.789482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:44.789617 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:44.790064 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.791158 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.792403 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:44.793081 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:44.793218 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:44.794396 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:44.794402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:44.794407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:44.794429 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:44.794483 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:44.794506 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:44.794611 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:44.794766 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 00:59:44.794880 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 00:59:44.797188 22572 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260929 00:59:44.797688 22572 leader_election.cc:302] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1cd27ec511a49dab54cc2cd1f9ff3cf; no voters:
I20260929 00:59:44.798252 22572 leader_election.cc:288] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:44.798355 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:44.799100 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [term 1 LEADER]: Becoming Leader. State: Replica: b1cd27ec511a49dab54cc2cd1f9ff3cf, State: Running, Role: LEADER
I20260929 00:59:44.799649 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } }
I20260929 00:59:44.800444 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:44.802716 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260929 00:59:44.803162 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:44.803965 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1cd27ec511a49dab54cc2cd1f9ff3cf. Latest consensus state: current_term: 1 leader_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cd27ec511a49dab54cc2cd1f9ff3cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34799 } } }
I20260929 00:59:44.804405 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:44.804888 22590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:44.811117 22590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:44.819953 22590 catalog_manager.cc:1347] Generated new cluster ID: c498eca4ebea45dfa22cd5cfa9b1dd67
I20260929 00:59:44.820067 22590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:44.831317 22590 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:44.833129 22590 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:44.846906 22590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1cd27ec511a49dab54cc2cd1f9ff3cf: Generated new TSK 0
I20260929 00:59:44.847820 22590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:44.861418 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22505
W20260929 00:59:44.862519 22567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50953
W20260929 00:59:44.863238 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:44.863366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22505
2026-09-29T00:59:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:44 dist-test-slave-pzsx krb5kdc[22603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:44 dist-test-slave-pzsx krb5kdc[22603](info): set up 2 sockets
Sep 29 00:59:44 dist-test-slave-pzsx krb5kdc[22603](info): commencing operation
krb5kdc: starting...
W20260929 00:59:45.973045 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:46 dist-test-slave-pzsx krb5kdc[22603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:46.228019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41655
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:46.558719 22616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:46.559010 22616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:46.559137 22616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:46.566356 22616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:46.566479 22616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:46.566525 22616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:46.566622 22616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:46.566694 22616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:46.577044 22616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:46.578311 22616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:46.579617 22616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:46.592867 22621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:46.592937 22624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:46.592891 22622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:46.593607 22616 server_base.cc:1034] running on GCE node
I20260929 00:59:46.594393 22616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:46.595849 22616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:46.597086 22616 hybrid_clock.cc:648] HybridClock initialized: now 1790643586597057 us; error 50 us; skew 500 ppm
I20260929 00:59:46.597446 22616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:46 dist-test-slave-pzsx krb5kdc[22603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:46.604480 22616 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:46.606076 22616 webserver.cc:460] Webserver started at http://127.0.97.62:36769/ using document root <none> and password file <none>
I20260929 00:59:46.606773 22616 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:46.606927 22616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:46.607234 22616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:46.610302 22616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "defe393e93424c41b8604764d2e4e314"
format_stamp: "Formatted at 2026-09-29 00:59:46 on dist-test-slave-pzsx"
server_key: "16cd10488d05d7f29e0ff41e7d590b98"
server_key_iv: "96437f74b3ad5b3779c2e7cf60c21d3b"
server_key_version: "encryptionkey@0"
I20260929 00:59:46.611068 22616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "defe393e93424c41b8604764d2e4e314"
format_stamp: "Formatted at 2026-09-29 00:59:46 on dist-test-slave-pzsx"
server_key: "16cd10488d05d7f29e0ff41e7d590b98"
server_key_iv: "96437f74b3ad5b3779c2e7cf60c21d3b"
server_key_version: "encryptionkey@0"
I20260929 00:59:46.616256 22616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:59:46.619650 22631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:46.621037 22616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 00:59:46.621232 22616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "defe393e93424c41b8604764d2e4e314"
format_stamp: "Formatted at 2026-09-29 00:59:46 on dist-test-slave-pzsx"
server_key: "16cd10488d05d7f29e0ff41e7d590b98"
server_key_iv: "96437f74b3ad5b3779c2e7cf60c21d3b"
server_key_version: "encryptionkey@0"
I20260929 00:59:46.621440 22616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:46.627146 22616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:46.631095 22616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:46.631407 22616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:46.649976 22616 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41655
I20260929 00:59:46.650023 22682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41655 every 8 connection(s)
I20260929 00:59:46.651379 22616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:46.652781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22616
I20260929 00:59:46.652948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:46.653479 388 external_mini_cluster.cc:1428] Setting key 3ce73a62a72ffdd8b425de34577321b2
I20260929 00:59:46.656059 22683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:46.661302 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:46.655245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36489 (local address 127.0.97.62:41655)
0929 00:59:46.656010 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:46.656022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:46.656065 (+ 43us) server_negotiation.cc:407] Connection header received
0929 00:59:46.657800 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:46.657822 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:46.658150 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:46.658571 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:46.659634 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:46.659639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:46.659652 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:46.659675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:46.659742 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:46.659760 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:46.660011 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:46.660161 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 00:59:46.660320 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":334,"threads_started":1}
I20260929 00:59:46.664907 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:46.667557 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314: Bootstrap starting.
I20260929 00:59:46.671408 22683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:46.671841 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:46.669308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45409 (local address 127.0.97.62:41655)
0929 00:59:46.669521 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:46.669526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:46.669868 (+ 342us) server_negotiation.cc:407] Connection header received
0929 00:59:46.670032 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:46.670035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:46.670083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:46.670171 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:46.671154 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:46.671159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:46.671162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 00:59:46.671175 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:46.671209 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 00:59:46.671223 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:46.671313 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:46.671471 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 00:59:46.671554 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 00:59:46.672802 22683 log.cc:824] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:46.675661 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314: No bootstrap required, opened a new log
I20260929 00:59:46.682932 22683 raft_consensus.cc:348] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } }
I20260929 00:59:46.683284 22683 raft_consensus.cc:374] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 00:59:46.683359 22683 raft_consensus.cc:729] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defe393e93424c41b8604764d2e4e314, State: Initialized, Role: FOLLOWER
I20260929 00:59:46.683862 22683 consensus_queue.cc:260] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } }
I20260929 00:59:46.684047 22683 raft_consensus.cc:388] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 00:59:46.684145 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 00:59:46.684269 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:46.686376 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } }
I20260929 00:59:46.686823 22683 leader_election.cc:302] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: defe393e93424c41b8604764d2e4e314; no voters:
I20260929 00:59:46.687337 22683 leader_election.cc:288] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 00:59:46.687485 22693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:46.688213 22693 raft_consensus.cc:686] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [term 1 LEADER]: Becoming Leader. State: Replica: defe393e93424c41b8604764d2e4e314, State: Running, Role: LEADER
I20260929 00:59:46.688798 22693 consensus_queue.cc:237] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } }
I20260929 00:59:46.689486 22683 sys_catalog.cc:564] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:46.694144 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader defe393e93424c41b8604764d2e4e314. Latest consensus state: current_term: 1 leader_uuid: "defe393e93424c41b8604764d2e4e314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } } }
I20260929 00:59:46.694175 22694 sys_catalog.cc:455] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "defe393e93424c41b8604764d2e4e314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defe393e93424c41b8604764d2e4e314" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41655 } } }
I20260929 00:59:46.694587 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:46.694638 22694 sys_catalog.cc:458] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:46.695490 22703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:46.700763 22703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:46.709676 22703 catalog_manager.cc:1347] Generated new cluster ID: 9cd8944486e6452fab6a159750bc4aa2
I20260929 00:59:46.709784 22703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:46.724802 22703 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:46.725247 22703 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:46.735862 22703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P defe393e93424c41b8604764d2e4e314: Generated new TSK 0
I20260929 00:59:46.736783 22703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:46.751768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22616
W20260929 00:59:46.753177 22678 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36489
W20260929 00:59:46.753875 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22616: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 00:59:46.754022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22616
2026-09-29T00:59:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1890 ms)
[----------] 18 tests from AuthTokenIssuingTest (34439 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:46 dist-test-slave-pzsx krb5kdc[22714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:46 dist-test-slave-pzsx krb5kdc[22714](info): set up 2 sockets
Sep 29 00:59:46 dist-test-slave-pzsx krb5kdc[22714](info): commencing operation
krb5kdc: starting...
W20260929 00:59:47.857225 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:48 dist-test-slave-pzsx krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:48.106186 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43295
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:48.428270 22727 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:48.428723 22727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:48.428839 22727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:48.428995 22727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:48.436223 22727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:48.436306 22727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:48.436414 22727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:48.436544 22727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:48.436671 22727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:48.446354 22727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:48.447655 22727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:48.449018 22727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:48.460987 22732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:48.461017 22735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:48.461102 22733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:48.461974 22727 server_base.cc:1034] running on GCE node
I20260929 00:59:48.462755 22727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:48.464146 22727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:48.465399 22727 hybrid_clock.cc:648] HybridClock initialized: now 1790643588465349 us; error 28 us; skew 500 ppm
I20260929 00:59:48.465755 22727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:48 dist-test-slave-pzsx krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:48.468966 22727 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:48.470443 22727 webserver.cc:460] Webserver started at http://127.0.97.62:34991/ using document root <none> and password file <none>
I20260929 00:59:48.471139 22727 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:48.471277 22727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:48.471606 22727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:48.474550 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "6ac25d4412ca48bea37ec0d0104ef114"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "6bc3888a9f11c7ea3e21d39848066c12"
server_key_iv: "e10053f98254462e5b10b348376b28c0"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.475325 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "6ac25d4412ca48bea37ec0d0104ef114"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "6bc3888a9f11c7ea3e21d39848066c12"
server_key_iv: "e10053f98254462e5b10b348376b28c0"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.480365 22727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 00:59:48.483659 22742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:48.485041 22727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:48.485234 22727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "6ac25d4412ca48bea37ec0d0104ef114"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "6bc3888a9f11c7ea3e21d39848066c12"
server_key_iv: "e10053f98254462e5b10b348376b28c0"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.485456 22727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:48.511220 22727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:48.515671 22727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:48.515967 22727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:48.534461 22727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43295
I20260929 00:59:48.534484 22793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43295 every 8 connection(s)
I20260929 00:59:48.535815 22727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:48.539520 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22727
I20260929 00:59:48.539743 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:48.539781 22794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:48.540612 388 external_mini_cluster.cc:1428] Setting key 41e9a2a0b53bedc0140bf9b2622c4638
I20260929 00:59:48.549861 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:48 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 00:59:48.575584 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.553728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.0.97.62:43295)
0929 00:59:48.555688 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:48.555694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:48.555736 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:59:48.557953 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:48.557959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:48.558188 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:48.558351 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:48.559957 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:48.561221 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:48.563466 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:48.564002 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:48.564128 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:48.568178 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:48.568191 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:48.568221 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:48.568267 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:48.571094 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:48.571806 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:48.571816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:48.571821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:48.571966 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:48.572369 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:48.572373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:48.572378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:48.572940 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:48.573309 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:48.573864 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 00:59:48.574197 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1770,"spinlock_wait_cycles":22400,"thread_start_us":1526,"threads_started":1}
I20260929 00:59:48.575641 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.553728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43295 (local address 127.0.0.1:51168)
0929 00:59:48.554702 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.554753 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:48.557582 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:48.558444 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:48.558454 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:48.559241 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:48.559743 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:48.559769 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:48.561387 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:48.561394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:48.563320 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:48.563338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:48.564263 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:48.564281 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:48.564492 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:48.565203 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:48.565245 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:48.567931 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:48.571244 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:48.571254 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:48.571266 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:48.571676 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:48.572084 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:48.572091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:48.572097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:48.572252 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:48.573164 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:48.573170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:48.573477 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 00:59:48.573749 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 00:59:48.574317 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":149,"thread_start_us":428,"threads_started":1}
I20260929 00:59:48.578761 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:48.579921 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.579555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43564)
0929 00:59:48.579715 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.579808 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 00:59:48.580185 22745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
I20260929 00:59:48.581763 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.581433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43566)
0929 00:59:48.581574 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.581657 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 00:59:48.581977 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 00:59:48.599802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37615
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:48.631273 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } attempt: 1
I20260929 00:59:48.632711 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.632170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43578)
0929 00:59:48.632378 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.632469 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 00:59:48.633762 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.633414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43592)
0929 00:59:48.633560 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.633638 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 00:59:48.633960 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
I20260929 00:59:48.702282 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } attempt: 2
I20260929 00:59:48.703481 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.702979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43606)
0929 00:59:48.703239 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.703336 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 00:59:48.704548 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.704123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43610)
0929 00:59:48.704270 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.704362 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:59:48.704749 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
I20260929 00:59:48.796111 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } attempt: 3
I20260929 00:59:48.797264 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.796821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43614)
0929 00:59:48.797034 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.797131 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 00:59:48.798259 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.797923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43626)
0929 00:59:48.798065 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.798138 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:59:48.798452 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
W20260929 00:59:48.921985 22801 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:48.922358 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:48.922490 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:48.922660 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:48.929834 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:48.929956 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:48.930095 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:48.930186 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:48.930280 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:48.939807 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:48.941080 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:48.941696 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } attempt: 4
I20260929 00:59:48.942514 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:48.942910 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.942325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43640)
0929 00:59:48.942665 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.942781 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 00:59:48.943919 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:48.943549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43656)
0929 00:59:48.943695 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:48.943784 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 00:59:48.944141 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37615: connect: Connection refused (error 111)
W20260929 00:59:48.954320 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:48.954435 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:48.954435 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:48.955261 22801 server_base.cc:1034] running on GCE node
I20260929 00:59:48.955993 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:48.957387 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:48.958623 22801 hybrid_clock.cc:648] HybridClock initialized: now 1790643588958546 us; error 54 us; skew 500 ppm
I20260929 00:59:48.958952 22801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:48 dist-test-slave-pzsx krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:48.962092 22801 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 00:59:48.963397 22801 webserver.cc:460] Webserver started at http://127.0.97.61:38729/ using document root <none> and password file <none>
I20260929 00:59:48.964021 22801 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:48.964102 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:48.964358 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:48.967152 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data/instance:
uuid: "0128ff85f0f14b74ba1349cd7559f102"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "c90aee70542a8b389134c72e9283b60d"
server_key_iv: "2825963c71f96726da2888f10fd653ff"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.967864 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal/instance:
uuid: "0128ff85f0f14b74ba1349cd7559f102"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "c90aee70542a8b389134c72e9283b60d"
server_key_iv: "2825963c71f96726da2888f10fd653ff"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.972702 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 00:59:48.975827 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:48.977124 22801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:48.977252 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "0128ff85f0f14b74ba1349cd7559f102"
format_stamp: "Formatted at 2026-09-29 00:59:48 on dist-test-slave-pzsx"
server_key: "c90aee70542a8b389134c72e9283b60d"
server_key_iv: "2825963c71f96726da2888f10fd653ff"
server_key_version: "encryptionkey@0"
I20260929 00:59:48.977429 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:48.990854 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:48.994525 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:48.994731 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:49.013048 22801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37615
I20260929 00:59:49.013087 22867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37615 every 8 connection(s)
I20260929 00:59:49.014600 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
I20260929 00:59:49.018123 22868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:49.023414 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22801
I20260929 00:59:49.023594 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-1/wal/instance
I20260929 00:59:49.024094 388 external_mini_cluster.cc:1428] Setting key e320c45a7e00a112bb1eed04b8a99c27
I20260929 00:59:49.025441 22868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:49.047120 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.028672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.0.97.62:43295)
0929 00:59:49.028907 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.028916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.030096 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 00:59:49.032916 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.032921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.032987 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.033077 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.035039 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.035823 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.037407 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.037833 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.037994 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.041336 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.041341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.041344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.041375 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.044088 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.044940 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.044950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.044953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.045065 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.045478 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.045483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.045486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.045769 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.045925 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.046595 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.046813 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 00:59:49.048094 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.029081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43295 (local address 127.0.0.1:51170)
0929 00:59:49.029898 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.029947 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.032604 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.033335 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.033348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.034126 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.034771 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.034789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.036006 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.036017 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.037254 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.037267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.038089 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.038096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.038325 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.039005 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.039026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.041116 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.044236 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.044245 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.044260 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.044811 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.045178 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.045183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.045187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.045360 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.045902 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.045910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.046242 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.046493 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.046966 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260929 00:59:49.049880 22868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:49.067418 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.050732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43670)
0929 00:59:49.050910 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.050929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.051035 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.052017 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.052025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.052566 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.052784 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.052795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.054164 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.054172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.055635 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.055652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.056469 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.056478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.056717 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.057379 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.057407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.060360 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.064615 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.064620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.064624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.065194 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.065655 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.065659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.065663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.065801 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.066469 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.066473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.066611 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.066757 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.067078 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 00:59:49.067968 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.050921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.97.61:37615)
0929 00:59:49.051613 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.051619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.051641 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:59:49.051680 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.051686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.051843 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.051955 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.052956 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.054005 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.055772 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.056278 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.056369 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.060597 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.060607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.060620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.060645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.064457 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.065342 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.065354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.065360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.065505 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.065973 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.065978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.065983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.066315 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.066543 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.066929 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.067239 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":367,"threads_started":1}
I20260929 00:59:49.070402 22868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:49.071491 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.071051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40554)
0929 00:59:49.071244 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.071364 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 00:59:49.071820 22820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.073366 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.073037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40568)
0929 00:59:49.073193 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.073263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 00:59:49.073673 22868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 00:59:49.085268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45277
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:49.122985 22868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 1
I20260929 00:59:49.124128 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.123676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40584)
0929 00:59:49.123891 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.123975 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 00:59:49.125100 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.124744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40590)
0929 00:59:49.124910 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.124998 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 00:59:49.125319 22868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.193521 22868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 2
I20260929 00:59:49.194615 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.194158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40602)
0929 00:59:49.194361 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.194465 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 00:59:49.195556 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.195238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40606)
0929 00:59:49.195385 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.195452 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 00:59:49.195758 22868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.243410 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } attempt: 5
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:49.258574 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.244207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43674)
0929 00:59:49.244433 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.244460 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.244667 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.245101 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.245105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.245497 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.245662 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.245670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.247075 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.247079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.247898 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.247908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.248904 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.248908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.249089 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.249745 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.249762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.251961 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.256165 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.256169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.256171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.256612 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.257058 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.257061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.257063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.257189 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.257893 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.257896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.258029 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.258136 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.258333 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 00:59:49.258821 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.244365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.97.61:37615)
0929 00:59:49.244575 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.244581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.244605 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:59:49.244817 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.244822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.244921 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.245033 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.245831 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.246926 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.248058 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.248774 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.248897 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.252173 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.252183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.252189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.252235 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.256018 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.256745 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.256752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.256758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.256931 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.257306 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.257313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.257319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.257764 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.257934 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.258282 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.258521 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 00:59:49.259827 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:49.260874 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.260407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40620)
0929 00:59:49.260609 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.260699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 00:59:49.261837 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.261452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40634)
0929 00:59:49.261631 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.261720 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 00:59:49.262038 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.286969 22868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 3
I20260929 00:59:49.288056 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.287597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40636)
0929 00:59:49.287816 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.287899 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 00:59:49.289223 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.288768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40642)
0929 00:59:49.289021 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.289111 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260929 00:59:49.289474 22868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.313279 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 1
I20260929 00:59:49.314563 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.313972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40652)
0929 00:59:49.314234 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.314354 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 00:59:49.315622 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.315216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40666)
0929 00:59:49.315397 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.315485 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 00:59:49.315856 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.384115 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 2
I20260929 00:59:49.385455 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.384885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40680)
0929 00:59:49.385146 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.385260 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 00:59:49.386672 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.386210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40686)
0929 00:59:49.386441 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.386533 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 00:59:49.386946 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
W20260929 00:59:49.412122 22875 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:49.412483 22875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:49.412636 22875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:49.412719 22875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:49.419703 22875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:49.419783 22875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:49.419858 22875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:49.419901 22875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:49.419953 22875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:49.429621 22875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43295,127.0.97.61:37615,127.0.97.60:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:49.430879 22875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:49.432149 22875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:49.432725 22868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 4
I20260929 00:59:49.433928 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.433341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40694)
0929 00:59:49.433658 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.433776 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 00:59:49.434876 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.434565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40708)
0929 00:59:49.434705 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.434776 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:59:49.435102 22868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
W20260929 00:59:49.444362 22883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:49.444433 22880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:49.444454 22881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:49.445480 22875 server_base.cc:1034] running on GCE node
I20260929 00:59:49.446367 22875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:49.447860 22875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:49.449126 22875 hybrid_clock.cc:648] HybridClock initialized: now 1790643589449065 us; error 33 us; skew 500 ppm
I20260929 00:59:49.449473 22875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:49.452477 22875 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 00:59:49.454028 22875 webserver.cc:460] Webserver started at http://127.0.97.60:42285/ using document root <none> and password file <none>
I20260929 00:59:49.454669 22875 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:49.454761 22875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:49.455020 22875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:49.457988 22875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data/instance:
uuid: "5ec65313c50240bb936f3d0539d67663"
format_stamp: "Formatted at 2026-09-29 00:59:49 on dist-test-slave-pzsx"
server_key: "6eaebc0cf91cd5d42f1c8d4e8ca04340"
server_key_iv: "e36f1ff5f044b05adac0ebc6b2aa4732"
server_key_version: "encryptionkey@0"
I20260929 00:59:49.458693 22875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal/instance:
uuid: "5ec65313c50240bb936f3d0539d67663"
format_stamp: "Formatted at 2026-09-29 00:59:49 on dist-test-slave-pzsx"
server_key: "6eaebc0cf91cd5d42f1c8d4e8ca04340"
server_key_iv: "e36f1ff5f044b05adac0ebc6b2aa4732"
server_key_version: "encryptionkey@0"
I20260929 00:59:49.463603 22875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:49.466969 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:49.468324 22875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 00:59:49.468483 22875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "5ec65313c50240bb936f3d0539d67663"
format_stamp: "Formatted at 2026-09-29 00:59:49 on dist-test-slave-pzsx"
server_key: "6eaebc0cf91cd5d42f1c8d4e8ca04340"
server_key_iv: "e36f1ff5f044b05adac0ebc6b2aa4732"
server_key_version: "encryptionkey@0"
I20260929 00:59:49.468732 22875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:49.487691 22875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:49.491750 22875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:49.492024 22875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:49.493175 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 3
I20260929 00:59:49.494411 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.493818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40714)
0929 00:59:49.494078 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.494186 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 00:59:49.495481 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.495013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40716)
0929 00:59:49.495244 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.495344 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 00:59:49.495687 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45277: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45277: connect: Connection refused (error 111)
I20260929 00:59:49.510774 22875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45277
I20260929 00:59:49.510790 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45277 every 8 connection(s)
I20260929 00:59:49.512081 22875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
I20260929 00:59:49.515622 22942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:49.519174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 22875
I20260929 00:59:49.519366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643237839665-388-0/minicluster-data/master-2/wal/instance
I20260929 00:59:49.519904 388 external_mini_cluster.cc:1428] Setting key 44849626d336fffe0536a764a68a696a
I20260929 00:59:49.523272 22942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:49.543705 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.527410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.0.97.62:43295)
0929 00:59:49.527583 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.527588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.528277 (+ 689us) server_negotiation.cc:407] Connection header received
0929 00:59:49.528467 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.528471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.528588 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.528667 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.530148 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.531200 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.533041 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.533653 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.533784 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.537697 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.537701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.537705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.537736 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.540667 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.541346 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.541355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.541359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.541534 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.541958 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.541963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.541966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.542292 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.542459 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.543122 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.543339 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:59:49.544770 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.527430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43295 (local address 127.0.0.1:51174)
0929 00:59:49.528098 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.528155 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.528341 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.528766 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.528772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.529597 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.529970 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.529986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.531351 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.531354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.532888 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.532900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.534042 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.534050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.534241 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.534919 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.534947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.537502 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.540812 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.540826 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.540836 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.541220 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.541642 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.541647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.541655 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.541857 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.542438 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.542445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.542802 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.543035 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.543470 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":37,"thread_start_us":288,"threads_started":1}
I20260929 00:59:49.544889 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.522080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.97.60:45277)
0929 00:59:49.522982 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.522997 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.523432 (+ 435us) server_negotiation.cc:407] Connection header received
0929 00:59:49.525390 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.525419 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.525905 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.526312 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.527950 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.530057 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.531676 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.532643 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.532846 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.537135 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.537147 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.537161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.537195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.540689 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.541584 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.541595 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.541599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.541786 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.542219 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.542224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.542227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.542839 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.543119 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.544167 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.544413 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":662,"thread_start_us":408,"threads_started":1}
I20260929 00:59:49.546440 22942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:49.560468 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.547237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37615 (local address 127.0.0.1:43676)
0929 00:59:49.547439 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.547453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.547536 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.547866 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.547870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.548276 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.548462 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.548474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.549474 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.549480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.550906 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.550917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.551649 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.551655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.551874 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.552482 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.552549 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.554854 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.558014 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.558020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.558024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.558559 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.558940 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.558945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.558948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.559123 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.559604 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.559609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.559786 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.559959 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.560227 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 00:59:49.560691 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.547196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.0.97.61:37615)
0929 00:59:49.547391 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.547399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.547530 (+ 131us) server_negotiation.cc:407] Connection header received
0929 00:59:49.547623 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.547628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.547711 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.547824 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.548620 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.549313 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.551047 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.551488 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.551628 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.555051 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.555060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.555066 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.555109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.557878 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.558678 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.558682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.558684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.558821 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.559231 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.559234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.559236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.559463 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.559574 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.560028 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.560274 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 00:59:49.561718 22942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:49.562947 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.547952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.0.97.62:43295)
0929 00:59:49.548094 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.548098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.548111 (+ 13us) server_negotiation.cc:407] Connection header received
0929 00:59:49.548148 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.548151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.548198 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.548271 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.549570 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.550657 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.551766 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.552148 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.552293 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.556802 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.556809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.556816 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.556875 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.559987 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.560903 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.560913 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.560915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.561033 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.561483 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.561488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.561491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.561794 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.562043 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.562413 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.562616 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:49.576272 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.562639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.97.60:45277)
0929 00:59:49.562789 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.562795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.562812 (+ 17us) server_negotiation.cc:407] Connection header received
0929 00:59:49.562872 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.562876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.562942 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.563039 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.563753 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.564736 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.566435 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.566903 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.567024 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.570336 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.570340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.570343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.570368 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.573478 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.574260 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.574263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.574266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.574383 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.574770 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.574775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.574779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.575114 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.575277 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.575608 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.575969 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":47}
I20260929 00:59:49.576269 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.562460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40732)
0929 00:59:49.562650 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.562669 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.562775 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.563104 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.563111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.563490 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.563612 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.563619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.564869 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.564877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.566286 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.566313 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.567049 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.567053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.567221 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.567719 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.567734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.570173 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.573609 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.573615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.573619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.574142 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.574531 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.574534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.574537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.574658 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.575244 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.575248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.575380 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.575528 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.575835 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":3}
I20260929 00:59:49.580096 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.564429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.0.97.61:37615)
0929 00:59:49.564598 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.564603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.565077 (+ 474us) server_negotiation.cc:407] Connection header received
0929 00:59:49.565207 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.565212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.565261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.565331 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.566778 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.567728 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.569422 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.569913 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.570001 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.574514 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.574520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.574525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.574557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.577605 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.578237 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.578243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.578249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.578426 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.578767 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.578772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.578776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.579102 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.579377 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.579525 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.579742 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 00:59:49.582369 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: Bootstrap starting.
I20260929 00:59:49.584933 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:49.585705 22942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:49.586722 22942 log.cc:824] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:49.589682 22942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: No bootstrap required, opened a new log
I20260929 00:59:49.587448 22769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ac25d4412ca48bea37ec0d0104ef114"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51190
I20260929 00:59:49.596633 22942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.596990 22942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec65313c50240bb936f3d0539d67663, State: Initialized, Role: FOLLOWER
I20260929 00:59:49.597412 22942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.598455 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.598812 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.599809 22942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:59:49.609859 22959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:49.609987 22959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:49.607398 22875 master_runner.cc:186] Error getting master registration for 127.0.97.62:43295: OK, instance_id { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" instance_seqno: 1790643588529154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:59:49.618502 22875 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:49.672887 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 4
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:59:49.687225 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.674249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.0.97.60:45277)
0929 00:59:49.674450 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.674456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.674481 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:59:49.674577 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.674582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.674666 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.674790 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.675580 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.676368 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.677629 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.678113 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.678271 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.681633 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.681638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.681641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.681670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.684452 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.685147 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.685152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.685155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.685288 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.685680 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.685686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.685689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.686026 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.686205 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.686581 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.686838 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 00:59:49.687232 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.673788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40734)
0929 00:59:49.674281 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.674328 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.674462 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.674832 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.674839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.675249 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.675425 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.675434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.676547 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.676552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.677481 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.677493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.678284 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.678290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.678490 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.679127 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.679146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.681452 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.684646 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.684650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.684654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.685016 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.685410 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.685416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.685420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.685566 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.686158 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.686163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.686325 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.686466 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.686801 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377}
I20260929 00:59:49.692222 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: Bootstrap starting.
I20260929 00:59:49.696163 22794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:49.697258 22794 log.cc:824] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:49.700060 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: No bootstrap required, opened a new log
I20260929 00:59:49.705475 22794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.705789 22794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac25d4412ca48bea37ec0d0104ef114, State: Initialized, Role: FOLLOWER
I20260929 00:59:49.706225 22794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.707171 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.707502 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.708202 22794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:49.708319 22769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 00:59:49.708547 22769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:49.711869 22769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.714310 22917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac25d4412ca48bea37ec0d0104ef114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ec65313c50240bb936f3d0539d67663"
I20260929 00:59:49.714800 22917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:49.718269 22917 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac25d4412ca48bea37ec0d0104ef114 in term 1.
I20260929 00:59:49.718652 22769 leader_election.cc:288] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [CANDIDATE]: Term 1 election: Requested vote from peers 0128ff85f0f14b74ba1349cd7559f102 (127.0.97.61:37615), 5ec65313c50240bb936f3d0539d67663 (127.0.97.60:45277)
I20260929 00:59:49.718909 22746 leader_election.cc:302] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec65313c50240bb936f3d0539d67663, 6ac25d4412ca48bea37ec0d0104ef114; no voters:
I20260929 00:59:49.719635 22962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:49.720536 22962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [term 1 LEADER]: Becoming Leader. State: Replica: 6ac25d4412ca48bea37ec0d0104ef114, State: Running, Role: LEADER
I20260929 00:59:49.721076 22962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.724983 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac25d4412ca48bea37ec0d0104ef114. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.725347 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:49.726676 22973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:49.734681 22868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } attempt: 5
W20260929 00:59:49.744279 22896 tablet.cc:2369] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 00:59:49 dist-test-slave-pzsx krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:59:49.755174 22973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:49.761291 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.736195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.97.60:45277)
0929 00:59:49.736421 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.736430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.736454 (+ 24us) server_negotiation.cc:407] Connection header received
0929 00:59:49.736545 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.736553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.736676 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.736813 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.738289 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.739813 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.742265 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.743198 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.743342 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.753449 (+ 10107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.753459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.753467 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.753519 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.757727 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.758592 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.758599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.758602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.758768 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.759254 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.759261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.759268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.759602 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.759746 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.760649 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.760903 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 00:59:49.763082 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.735244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45277 (local address 127.0.0.1:40742)
0929 00:59:49.735643 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:49.735663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:49.735814 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:49.737080 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:49.737088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:49.737758 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:49.738027 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.738043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.740047 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.740056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.742019 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:49.742038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.743633 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.743639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:49.744083 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.745181 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:49.745236 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:49.753253 (+ 8017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:49.757885 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.757890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.757894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.758444 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.758907 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:49.758912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:49.758915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:49.759089 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:49.759929 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:49.759933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:49.760096 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 00:59:49.760266 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 00:59:49.762670 (+ 2404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260929 00:59:49.769860 22868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: Bootstrap starting.
I20260929 00:59:49.773913 22868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:49.775244 22868 log.cc:824] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: Log is configured to *not* fsync() on all Append() calls
W20260929 00:59:49.768837 22748 tablet.cc:2369] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:49.791939 22868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: No bootstrap required, opened a new log
I20260929 00:59:49.803071 22868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.803558 22868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0128ff85f0f14b74ba1349cd7559f102, State: Initialized, Role: FOLLOWER
I20260929 00:59:49.804150 22868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } }
I20260929 00:59:49.811028 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.731960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.0.97.62:43295)
0929 00:59:49.732133 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.732140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.747086 (+ 14946us) server_negotiation.cc:407] Connection header received
0929 00:59:49.747537 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.747545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.747636 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.747749 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.773427 (+ 25678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.774589 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.780956 (+ 6367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.790243 (+ 9287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.790373 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.793423 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.793431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.793435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.793477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.797689 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.805518 (+ 7829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.805524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.805528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.805692 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.806854 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.806859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.806863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.807261 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.807507 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.810352 (+ 2845us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.810611 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:59:49.815223 22917 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [term 1 FOLLOWER]: Refusing update from remote peer 6ac25d4412ca48bea37ec0d0104ef114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:49.816167 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.734445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.0.97.61:37615)
0929 00:59:49.734671 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.734680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.759325 (+ 24645us) server_negotiation.cc:407] Connection header received
0929 00:59:49.760324 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.760329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.760394 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.760456 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.774744 (+ 14288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.775891 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.781295 (+ 5404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.781991 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.782080 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.786413 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.786419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.786424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.786470 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.790420 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.808045 (+ 17625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.808051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.808054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.808223 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.809383 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.809388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.809392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.809890 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.810119 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.815581 (+ 5462us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.815856 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:49.816398 22970 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:49.822523 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:49.747233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.97.60:45277)
0929 00:59:49.761758 (+ 14525us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:49.761763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:49.769381 (+ 7618us) server_negotiation.cc:407] Connection header received
0929 00:59:49.771201 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:49.771207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:49.771324 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:49.771394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:49.777689 (+ 6295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.778720 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.781988 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:49.782650 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:49.782752 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:49.788068 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:49.788076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:49.788081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:49.788120 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:49.799501 (+ 11381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.817412 (+ 17911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.817417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.817421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.817577 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:49.819610 (+ 2033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:49.819615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:49.819618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:49.820016 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:49.820318 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:49.821829 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0929 00:59:49.822108 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14415,"thread_start_us":25985,"threads_started":1}
I20260929 00:59:49.789261 22727 master_runner.cc:186] Error getting master registration for 127.0.97.61:37615: OK, instance_id { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" instance_seqno: 1790643589007751 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:59:49.826977 22727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:49.827270 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac25d4412ca48bea37ec0d0104ef114. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.827585 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.833184 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.833629 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.835515 22843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac25d4412ca48bea37ec0d0104ef114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0128ff85f0f14b74ba1349cd7559f102"
I20260929 00:59:49.835871 22842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:49.836045 22868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:49.839409 22973 catalog_manager.cc:1347] Generated new cluster ID: 7f431695884245d5ab338341c4206ab8
I20260929 00:59:49.839494 22973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:49.839998 22842 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [term 1 FOLLOWER]: Refusing update from remote peer 6ac25d4412ca48bea37ec0d0104ef114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:49.846163 22989 mvcc.cc:205] Tried to move back new op lower bound from 7334476143867748352 to 7334476143502880768. Current Snapshot: MvccSnapshot[applied={T|T < 7334476143867748352}]
I20260929 00:59:49.846681 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.847172 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:49.844419 22843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac25d4412ca48bea37ec0d0104ef114 in term 1.
I20260929 00:59:49.849745 22962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:49.864464 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.864763 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.869431 22973 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:49.871178 22973 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:49.892593 22973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114: Generated new TSK 0
I20260929 00:59:49.893639 22973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:49.897450 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.897897 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.898238 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac25d4412ca48bea37ec0d0104ef114. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.898522 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:49.899499 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ac25d4412ca48bea37ec0d0104ef114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac25d4412ca48bea37ec0d0104ef114" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43295 } } peers { permanent_uuid: "0128ff85f0f14b74ba1349cd7559f102" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37615 } } peers { permanent_uuid: "5ec65313c50240bb936f3d0539d67663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45277 } } }
I20260929 00:59:49.899842 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac25d4412ca48bea37ec0d0104ef114 [sys.catalog]: This master's current role is: LEADER
W20260929 00:59:50.035259 22798 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.010417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.0.97.62:43295)
0929 00:59:50.010630 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.010636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.032396 (+ 21760us) server_negotiation.cc:407] Connection header received
0929 00:59:50.033486 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.033494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.033605 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.033689 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.034984 (+ 1295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51216: BlockingRecv error: recv got EOF from 127.0.0.1:51216 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 00:59:50.039532 22872 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.017012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.0.97.61:37615)
0929 00:59:50.017275 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.017284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.033364 (+ 16080us) server_negotiation.cc:407] Connection header received
0929 00:59:50.037624 (+ 4260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.037631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.037755 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.037878 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.039317 (+ 1439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43698: BlockingRecv error: recv got EOF from 127.0.0.1:43698 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 00:59:50.042730 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.040608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.97.60:45277)
0929 00:59:50.040831 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.040836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.041334 (+ 498us) server_negotiation.cc:407] Connection header received
0929 00:59:50.041537 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.041543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.041628 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.041687 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.042503 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40758: BlockingRecv error: recv got EOF from 127.0.0.1:40758 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:59:50.060103 23015 catalog_manager.cc:1259] Loaded cluster ID: 7f431695884245d5ab338341c4206ab8
I20260929 00:59:50.060237 23015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: loading cluster ID for follower catalog manager: success
I20260929 00:59:50.064298 23015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: acquiring CA information for follower catalog manager: success
I20260929 00:59:50.066685 23015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0128ff85f0f14b74ba1349cd7559f102: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:59:50.099166 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.083094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.97.61:37615)
0929 00:59:50.083287 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.083293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.091467 (+ 8174us) server_negotiation.cc:407] Connection header received
0929 00:59:50.091979 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.091986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.092178 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.092310 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.093051 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.094179 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.096212 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.096939 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.097034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.097739 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.098376 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.098485 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.098728 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.098825 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 00:59:50.142835 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.133751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.0.97.60:45277)
0929 00:59:50.133958 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.133970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.140760 (+ 6790us) server_negotiation.cc:407] Connection header received
0929 00:59:50.141276 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.141282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.141426 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.141505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.142573 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40762: BlockingRecv error: recv got EOF from 127.0.0.1:40762 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:59:50.190979 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.176261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.97.61:37615)
0929 00:59:50.176470 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.176480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.183922 (+ 7442us) server_negotiation.cc:407] Connection header received
0929 00:59:50.184421 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.184427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.184569 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.184639 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.185331 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.186540 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.188565 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.189257 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.189394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.190088 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.190259 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.190323 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.190546 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.190638 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 00:59:50.206473 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.200444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.0.97.60:45277)
0929 00:59:50.200680 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.200686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.205059 (+ 4373us) server_negotiation.cc:407] Connection header received
0929 00:59:50.205239 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.205245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.205341 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.205450 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.206256 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40772: BlockingRecv error: recv got EOF from 127.0.0.1:40772 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 00:59:50.218283 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.211360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.0.97.61:37615)
0929 00:59:50.211522 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.211528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.211970 (+ 442us) server_negotiation.cc:407] Connection header received
0929 00:59:50.212108 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.212114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.212195 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.212295 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.212774 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.213947 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.215883 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.216635 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.216752 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.217368 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.217535 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.217630 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.217820 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.217897 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 00:59:50.232144 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.229970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.97.60:45277)
0929 00:59:50.230156 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.230162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.230711 (+ 549us) server_negotiation.cc:407] Connection header received
0929 00:59:50.230870 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.230876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.230972 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.231080 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.231908 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40780: BlockingRecv error: recv got EOF from 127.0.0.1:40780 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:59:50.247342 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.240095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.97.61:37615)
0929 00:59:50.240306 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.240312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.241122 (+ 810us) server_negotiation.cc:407] Connection header received
0929 00:59:50.241264 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.241270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.241358 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.241454 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.241983 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.243181 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.245119 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.245841 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.245967 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.246559 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.246718 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.246816 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.246979 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.247061 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260929 00:59:50.256150 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.253967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.0.97.60:45277)
0929 00:59:50.254246 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.254252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.254781 (+ 529us) server_negotiation.cc:407] Connection header received
0929 00:59:50.254943 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.254949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.255044 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.255149 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.255922 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40788: BlockingRecv error: recv got EOF from 127.0.0.1:40788 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 00:59:50.274447 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.267180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.0.97.61:37615)
0929 00:59:50.267373 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.267380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.267860 (+ 480us) server_negotiation.cc:407] Connection header received
0929 00:59:50.267993 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.267999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.268117 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.268211 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.268678 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.269825 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.272178 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.272911 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.273028 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.273599 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.273736 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.273807 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.274041 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.274127 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 00:59:50.282575 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.280486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.97.60:45277)
0929 00:59:50.280704 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.280710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.281172 (+ 462us) server_negotiation.cc:407] Connection header received
0929 00:59:50.281338 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.281344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.281436 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.281537 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.282316 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40792: BlockingRecv error: recv got EOF from 127.0.0.1:40792 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 00:59:50.308785 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.301794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.97.61:37615)
0929 00:59:50.301975 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.301982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.302504 (+ 522us) server_negotiation.cc:407] Connection header received
0929 00:59:50.302682 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.302689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.302789 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.302888 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.303371 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.304612 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.306546 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.307232 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.307364 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.307878 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.308009 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.308110 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.308289 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.308379 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 00:59:50.317948 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.315737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.0.97.60:45277)
0929 00:59:50.315918 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.315924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.316429 (+ 505us) server_negotiation.cc:407] Connection header received
0929 00:59:50.316611 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.316616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.316714 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.316821 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.317704 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40800: BlockingRecv error: recv got EOF from 127.0.0.1:40800 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 00:59:50.365653 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.358436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.0.97.61:37615)
0929 00:59:50.358635 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.358641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.359212 (+ 571us) server_negotiation.cc:407] Connection header received
0929 00:59:50.359438 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.359446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.359563 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.359686 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.360158 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.361384 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.363355 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.364105 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.364229 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.364752 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.364908 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.364996 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.365212 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.365309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 00:59:50.375587 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.373158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.0.97.60:45277)
0929 00:59:50.373355 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.373362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.374108 (+ 746us) server_negotiation.cc:407] Connection header received
0929 00:59:50.374263 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.374269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.374367 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.374478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.375316 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40812: BlockingRecv error: recv got EOF from 127.0.0.1:40812 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:59:50.458204 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.452067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.97.61:37615)
0929 00:59:50.452257 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.452261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.452588 (+ 327us) server_negotiation.cc:407] Connection header received
0929 00:59:50.452839 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.452843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.452920 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.453025 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.453496 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.454280 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.456275 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.456833 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.456926 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.457440 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.457544 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.457641 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.457807 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.457872 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 00:59:50.466840 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.464583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.0.97.60:45277)
0929 00:59:50.464840 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.464850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.465400 (+ 550us) server_negotiation.cc:407] Connection header received
0929 00:59:50.465654 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.465658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.465732 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.465843 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.466597 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40824: BlockingRecv error: recv got EOF from 127.0.0.1:40824 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:50.605209 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.598893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.97.61:37615)
0929 00:59:50.599210 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.599216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.599397 (+ 181us) server_negotiation.cc:407] Connection header received
0929 00:59:50.599615 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.599620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.599689 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.599770 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.600153 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.600929 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.602850 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.603660 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.603804 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.604322 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.604535 (+ 213us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.604636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.604786 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.604875 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260929 00:59:50.612203 22959 catalog_manager.cc:1259] Loaded cluster ID: 7f431695884245d5ab338341c4206ab8
I20260929 00:59:50.612320 22959 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: loading cluster ID for follower catalog manager: success
W20260929 00:59:50.612984 22945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:50.610912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.97.60:45277)
0929 00:59:50.611127 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.611134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.611498 (+ 364us) server_negotiation.cc:407] Connection header received
0929 00:59:50.611637 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.611641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.611708 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.611784 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:50.612727 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40838: BlockingRecv error: recv got EOF from 127.0.0.1:40838 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:50.615182 22959 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: acquiring CA information for follower catalog manager: success
I20260929 00:59:50.617043 22959 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ec65313c50240bb936f3d0539d67663: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:59:50.878433 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.873856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.0.97.61:37615)
0929 00:59:50.874093 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.874099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.874245 (+ 146us) server_negotiation.cc:407] Connection header received
0929 00:59:50.874410 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.874414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.874483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.874564 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.874928 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.875629 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.876941 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.877361 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.877451 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.877773 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.877876 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.877940 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.878060 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.878118 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 00:59:50.889788 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.884272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.97.60:45277)
0929 00:59:50.884451 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.884455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.884779 (+ 324us) server_negotiation.cc:407] Connection header received
0929 00:59:50.884920 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.884925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.885080 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.885181 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.885645 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.886342 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.887584 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.887971 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.888065 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.888626 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.889216 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.889331 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.889435 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.889528 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:59:50.908099 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.896671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.0.97.60:45277)
0929 00:59:50.896925 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.896934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.901335 (+ 4401us) server_negotiation.cc:407] Connection header received
0929 00:59:50.901594 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.901599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.901682 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.901739 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.902572 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.903676 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.905993 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.906504 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.906653 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.907256 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.907401 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.907495 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.907712 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.907802 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 00:59:50.908171 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.895874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.0.97.61:37615)
0929 00:59:50.896120 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.896130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.901868 (+ 5738us) server_negotiation.cc:407] Connection header received
0929 00:59:50.902092 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.902099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.902206 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.902276 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.903096 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.903891 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.906119 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.906753 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.906914 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.907375 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.907498 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.907559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.907798 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.907882 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:59:50.908377 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:50.895738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.0.97.62:43295)
0929 00:59:50.901450 (+ 5712us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:50.901457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:50.901477 (+ 20us) server_negotiation.cc:407] Connection header received
0929 00:59:50.901612 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:50.901618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:50.901830 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:50.902012 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:50.902823 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.903894 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.905543 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:50.905996 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:50.906122 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:50.906707 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:50.907176 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:50.907272 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:50.907406 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 00:59:50.907485 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5592,"thread_start_us":5500,"threads_started":1}
I20260929 00:59:50.913899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22727
I20260929 00:59:50.915113 22789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:51.026664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22727
I20260929 00:59:51.048005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22801
I20260929 00:59:51.049175 22863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:51.156915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22801
I20260929 00:59:51.177561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 22875
I20260929 00:59:51.178684 22937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:51.284192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 22875
2026-09-29T00:59:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4537 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 00:59:51 dist-test-slave-pzsx krb5kdc[23132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 00:59:51 dist-test-slave-pzsx krb5kdc[23132](info): set up 2 sockets
Sep 29 00:59:51 dist-test-slave-pzsx krb5kdc[23132](info): commencing operation
krb5kdc: starting...
W20260929 00:59:52.401208 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/test-user.keytab.
Sep 29 00:59:52 dist-test-slave-pzsx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T00:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T00:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 00:59:52.653957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34113
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 00:59:52.979048 23145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:52.979435 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:52.979557 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:52.979696 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:52.986838 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:52.986958 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:52.987061 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:52.987142 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:52.987231 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:52.996634 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:52.997903 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:52.999181 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:53.010705 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:53.010644 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:53.010912 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:53.011543 23145 server_base.cc:1034] running on GCE node
I20260929 00:59:53.012326 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:53.013679 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:53.014914 23145 hybrid_clock.cc:648] HybridClock initialized: now 1790643593014849 us; error 31 us; skew 500 ppm
I20260929 00:59:53.015257 23145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:53.022279 23145 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 00:59:53.023635 23145 webserver.cc:460] Webserver started at http://127.0.97.62:40807/ using document root <none> and password file <none>
I20260929 00:59:53.024327 23145 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:53.024453 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:53.024780 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:53.027666 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data/instance:
uuid: "355a2ac63aab484c8000e4ac4a8cc288"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "a6a178481cbde3f261d4fce24b70601e"
server_key_iv: "d77a0f6dfdd7b2d301257ee40e17e6b5"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.028448 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance:
uuid: "355a2ac63aab484c8000e4ac4a8cc288"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "a6a178481cbde3f261d4fce24b70601e"
server_key_iv: "d77a0f6dfdd7b2d301257ee40e17e6b5"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.033571 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 00:59:53.036799 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:53.038044 23145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 00:59:53.038225 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal
uuid: "355a2ac63aab484c8000e4ac4a8cc288"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "a6a178481cbde3f261d4fce24b70601e"
server_key_iv: "d77a0f6dfdd7b2d301257ee40e17e6b5"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.038434 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:53.054170 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:53.057847 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:53.058101 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:53.075790 23145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34113
I20260929 00:59:53.075816 23211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34113 every 8 connection(s)
I20260929 00:59:53.077116 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/data/info.pb
I20260929 00:59:53.080616 23212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:53.086472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 23145
I20260929 00:59:53.086678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-0/wal/instance
I20260929 00:59:53.087097 388 external_mini_cluster.cc:1428] Setting key 8c8b52623697c9d84bfed6c8615a4a34
I20260929 00:59:53.086894 23212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:53.111760 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.089831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34113 (local address 127.0.0.1:45674)
0929 00:59:53.091008 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.091060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:53.093732 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:53.094571 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:53.094581 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:53.095436 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:53.095931 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.095957 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.098141 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.098145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.099215 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.099227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.100568 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.100574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.100691 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.101352 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:53.101374 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:53.103692 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:53.107854 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.107863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.107871 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.108244 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.108731 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.108734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.108736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.108878 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.109867 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:53.109873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:53.110075 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 00:59:53.110350 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 00:59:53.110751 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"spinlock_wait_cycles":11520,"thread_start_us":682,"threads_started":1}
I20260929 00:59:53.111816 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.089832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.0.97.62:34113)
0929 00:59:53.090674 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:53.090693 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:53.091196 (+ 503us) server_negotiation.cc:407] Connection header received
0929 00:59:53.094095 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:53.094101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:53.094331 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:53.094485 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:53.096148 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.097996 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.099355 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.100310 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.100524 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.103935 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:53.103947 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:53.103965 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:53.104005 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:53.107738 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.108371 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.108381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.108387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.108615 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.108988 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.108995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.109002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.109723 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:53.109924 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:53.110516 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 00:59:53.110962 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":565,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260929 00:59:53.114760 23212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:53.115906 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.115540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56948)
0929 00:59:53.115686 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.115787 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:59:53.116174 23164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 00:59:53.117745 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.117403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56960)
0929 00:59:53.117551 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.117620 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 00:59:53.117965 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 00:59:53.147256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33857
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:53.167352 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } attempt: 1
I20260929 00:59:53.168468 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.168030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56962)
0929 00:59:53.168225 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.168322 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 00:59:53.169507 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.169175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56968)
0929 00:59:53.169331 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.169392 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 00:59:53.169697 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
I20260929 00:59:53.237967 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } attempt: 2
I20260929 00:59:53.239215 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.238714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56970)
0929 00:59:53.238980 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.239069 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 00:59:53.240283 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.239945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56972)
0929 00:59:53.240087 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.240162 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:59:53.240482 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
I20260929 00:59:53.331766 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } attempt: 3
I20260929 00:59:53.333113 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.332663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:56986)
0929 00:59:53.332860 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.332957 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 00:59:53.334276 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.333815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57000)
0929 00:59:53.334073 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.334152 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260929 00:59:53.334463 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
W20260929 00:59:53.476596 23219 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:53.476979 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:53.477105 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:53.477245 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 00:59:53.477697 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } attempt: 4
I20260929 00:59:53.478863 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.478332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57014)
0929 00:59:53.478624 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.478722 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 00:59:53.479857 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.479518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57028)
0929 00:59:53.479656 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.479735 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 00:59:53.480049 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33857: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33857: connect: Connection refused (error 111)
W20260929 00:59:53.484843 23219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:53.484918 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:53.485066 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:53.485141 23219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:53.485251 23219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:53.494987 23219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:53.496316 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:53.497666 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 00:59:53.509392 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:53.509477 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:53.509423 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:53.510205 23219 server_base.cc:1034] running on GCE node
I20260929 00:59:53.511025 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:53.512336 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:53.513546 23219 hybrid_clock.cc:648] HybridClock initialized: now 1790643593513477 us; error 42 us; skew 500 ppm
I20260929 00:59:53.513887 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:53.516947 23219 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 00:59:53.518362 23219 webserver.cc:460] Webserver started at http://127.0.97.61:37013/ using document root <none> and password file <none>
I20260929 00:59:53.519048 23219 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:53.519176 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:53.519486 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:53.522424 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data/instance:
uuid: "6755fdc66c314a8798205e70f3239bbc"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "c54ef077ab52226f255cf1759376583e"
server_key_iv: "5eea029e866485deeb288be567f5dd10"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.523205 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal/instance:
uuid: "6755fdc66c314a8798205e70f3239bbc"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "c54ef077ab52226f255cf1759376583e"
server_key_iv: "5eea029e866485deeb288be567f5dd10"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.528226 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 00:59:53.531503 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:53.532789 23219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 00:59:53.532964 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal
uuid: "6755fdc66c314a8798205e70f3239bbc"
format_stamp: "Formatted at 2026-09-29 00:59:53 on dist-test-slave-pzsx"
server_key: "c54ef077ab52226f255cf1759376583e"
server_key_iv: "5eea029e866485deeb288be567f5dd10"
server_key_version: "encryptionkey@0"
I20260929 00:59:53.533174 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:53.551211 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:53.554998 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:53.555270 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:53.573220 23219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33857
I20260929 00:59:53.573239 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33857 every 8 connection(s)
I20260929 00:59:53.574537 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/data/info.pb
I20260929 00:59:53.578104 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:53.581740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 23219
I20260929 00:59:53.581933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-1/wal/instance
I20260929 00:59:53.582360 388 external_mini_cluster.cc:1428] Setting key ef64da5d817808450f76db5fb95c7214
I20260929 00:59:53.584530 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 00:59:53.604023 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.587177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.62:34113)
0929 00:59:53.587422 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:53.587431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:53.588491 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 00:59:53.590369 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:53.590374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:53.590467 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:53.590578 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:53.591998 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.592748 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.594228 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.594666 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.594816 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.598224 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:53.598229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:53.598232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:53.598265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:53.601047 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.601766 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.601773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.601777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.601913 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.602322 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.602329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.602334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.602612 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:53.602769 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:53.603471 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 00:59:53.603689 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 00:59:53.605180 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.587533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34113 (local address 127.0.0.1:45676)
0929 00:59:53.588317 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.588364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:53.590094 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:53.590808 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:53.590815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:53.591312 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:53.591833 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.591850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.592910 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.592914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.594070 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.594082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.594899 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.594907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.595139 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.595822 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:53.595844 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:53.597998 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:53.601186 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.601195 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.601218 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.601640 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.602031 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.602036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.602041 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.602209 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.602750 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:53.602757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:53.603100 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 00:59:53.603377 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 00:59:53.603899 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 00:59:53.607139 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:53.623662 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.608156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.0.97.61:33857)
0929 00:59:53.608898 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:53.608907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:53.608940 (+ 33us) server_negotiation.cc:407] Connection header received
0929 00:59:53.608998 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:53.609008 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:53.609232 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:53.609383 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:53.610053 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.611740 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.613192 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.614078 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.614217 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.617933 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:53.617944 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:53.617958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:53.617986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:53.620769 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.621590 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.621596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.621598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.621706 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.622095 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.622098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.622100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.622429 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:53.622583 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:53.623023 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 00:59:53.623352 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":564,"thread_start_us":427,"threads_started":1}
I20260929 00:59:53.623745 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.607994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57034)
0929 00:59:53.608181 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.608201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:53.608319 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:53.609385 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:53.609389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:53.609749 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:53.609881 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.609889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.611883 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.611887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.613019 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.613033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.614222 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.614228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.614435 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.615196 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:53.615223 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:53.617717 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:53.620895 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.620904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.620910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.621465 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.621808 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.621815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.621821 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.621996 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.622565 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:53.622572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:53.622736 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 00:59:53.622891 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 00:59:53.623224 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:59:53.626283 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:53.627461 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.626906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60680)
0929 00:59:53.627111 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.627265 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 00:59:53.627725 23235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.629293 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.628969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60682)
0929 00:59:53.629120 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.629194 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:53.629531 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 00:59:53.641420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
/tmp/dist-test-taskHiPz2y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33635
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 00:59:53.678881 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 1
I20260929 00:59:53.680109 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.679563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60692)
0929 00:59:53.679804 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.679917 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 00:59:53.681137 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.680769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60708)
0929 00:59:53.680925 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.681016 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 00:59:53.681350 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.749591 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 2
I20260929 00:59:53.750826 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.750291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60714)
0929 00:59:53.750517 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.750638 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 00:59:53.751844 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.751482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60726)
0929 00:59:53.751628 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.751712 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:53.752060 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.779412 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } attempt: 5
Sep 29 00:59:53 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:53.792591 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.780288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.97.61:33857)
0929 00:59:53.780506 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:53.780530 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:53.780551 (+ 21us) server_negotiation.cc:407] Connection header received
0929 00:59:53.780732 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:53.780736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:53.780807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:53.780914 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:53.781709 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.782412 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.783586 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.784022 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.784161 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.787443 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:53.787449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:53.787452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:53.787483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:53.790075 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.790735 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.790743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.790751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.790873 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:53.791240 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:53.791246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:53.791249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:53.791519 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:53.791674 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:53.792057 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 00:59:53.792259 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 00:59:53.792610 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.780169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57050)
0929 00:59:53.780370 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.780395 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:53.780556 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:53.780951 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:53.780956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:53.781376 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:53.781553 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.781562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.782560 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.782565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.783436 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:53.783446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:53.784180 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:53.784186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:53.784370 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:53.785027 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:53.785044 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:53.787234 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:53.790215 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.790220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.790224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.790608 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.790985 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:53.790990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:53.790994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:53.791131 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:53.791637 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:53.791642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:53.791795 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 00:59:53.791940 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 00:59:53.792227 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 00:59:53.794003 23212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:53.794994 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.794617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60728)
0929 00:59:53.794777 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.794858 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 00:59:53.795898 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.795587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60736)
0929 00:59:53.795722 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.795810 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:53.796084 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.843279 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 3
I20260929 00:59:53.844621 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.844041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60748)
0929 00:59:53.844261 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.844384 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 00:59:53.845824 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.845258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60752)
0929 00:59:53.845536 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.845652 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260929 00:59:53.846060 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.847282 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 1
I20260929 00:59:53.848196 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.847794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60760)
0929 00:59:53.847967 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.848048 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 00:59:53.849197 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.848893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60774)
0929 00:59:53.849035 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.849103 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 00:59:53.849403 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:53.917629 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 2
I20260929 00:59:53.918756 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.918279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60780)
0929 00:59:53.918502 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.918596 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 00:59:53.919890 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.919489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60784)
0929 00:59:53.919653 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.919749 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 00:59:53.920102 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
W20260929 00:59:53.968449 23293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 00:59:53.968778 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 00:59:53.968850 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 00:59:53.968933 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 00:59:53.976131 23293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 00:59:53.976221 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 00:59:53.976284 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 00:59:53.976333 23293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 00:59:53.976392 23293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 00:59:53.986007 23293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34113,127.0.97.61:33857,127.0.97.60:33635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 00:59:53.987164 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 00:59:53.988348 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 00:59:53.989315 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 4
I20260929 00:59:53.990512 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.989967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60792)
0929 00:59:53.990222 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.990325 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 00:59:53.991493 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:53.991139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60800)
0929 00:59:53.991287 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:53.991369 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 00:59:53.991693 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
W20260929 00:59:54.000243 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:54.000092 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 00:59:54.000212 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 00:59:54.001312 23293 server_base.cc:1034] running on GCE node
I20260929 00:59:54.002002 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 00:59:54.003324 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 00:59:54.004594 23293 hybrid_clock.cc:648] HybridClock initialized: now 1790643594004543 us; error 48 us; skew 500 ppm
I20260929 00:59:54.004913 23293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 00:59:54.007967 23293 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 00:59:54.009397 23293 webserver.cc:460] Webserver started at http://127.0.97.60:41189/ using document root <none> and password file <none>
I20260929 00:59:54.010094 23293 fs_manager.cc:359] Metadata directory not provided
I20260929 00:59:54.010223 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 00:59:54.010542 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 00:59:54.013513 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data/instance:
uuid: "bd02c7fcc6774b98b6e097f95cb85c6f"
format_stamp: "Formatted at 2026-09-29 00:59:54 on dist-test-slave-pzsx"
server_key: "95c657cf0f36c28f4b4985ef44df49c7"
server_key_iv: "46be00e6707ec51db5c9998ee1f59b30"
server_key_version: "encryptionkey@0"
I20260929 00:59:54.014276 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal/instance:
uuid: "bd02c7fcc6774b98b6e097f95cb85c6f"
format_stamp: "Formatted at 2026-09-29 00:59:54 on dist-test-slave-pzsx"
server_key: "95c657cf0f36c28f4b4985ef44df49c7"
server_key_iv: "46be00e6707ec51db5c9998ee1f59b30"
server_key_version: "encryptionkey@0"
I20260929 00:59:54.019337 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 00:59:54.022693 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 00:59:54.023953 23293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 00:59:54.024138 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal
uuid: "bd02c7fcc6774b98b6e097f95cb85c6f"
format_stamp: "Formatted at 2026-09-29 00:59:54 on dist-test-slave-pzsx"
server_key: "95c657cf0f36c28f4b4985ef44df49c7"
server_key_iv: "46be00e6707ec51db5c9998ee1f59b30"
server_key_version: "encryptionkey@0"
I20260929 00:59:54.024364 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 00:59:54.026305 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 3
I20260929 00:59:54.027343 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.026883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60810)
0929 00:59:54.027100 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.027183 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 00:59:54.028311 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.027940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60812)
0929 00:59:54.028146 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.028212 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 00:59:54.028502 23212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33635: connect: Connection refused (error 111)
I20260929 00:59:54.046401 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 00:59:54.050390 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 00:59:54.050663 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 00:59:54.069623 23293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33635
I20260929 00:59:54.069598 23359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33635 every 8 connection(s)
I20260929 00:59:54.070984 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/data/info.pb
I20260929 00:59:54.075924 23360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 00:59:54.075985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu as pid 23293
I20260929 00:59:54.076159 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiPz2y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643237839665-388-0/minicluster-data/master-2/wal/instance
I20260929 00:59:54.076588 388 external_mini_cluster.cc:1428] Setting key bfec7de5251ce8a56163afc56ef563ed
I20260929 00:59:54.085062 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:59:54.106243 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.087767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.0.97.62:34113)
0929 00:59:54.087985 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.087994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.088756 (+ 762us) server_negotiation.cc:407] Connection header received
0929 00:59:54.090763 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.090772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.090896 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.091053 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.092370 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.093499 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.095724 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.096342 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.096454 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.100022 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.100027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.100030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.100062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.102909 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.103672 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.103680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.103685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.103845 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.104357 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.104364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.104368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.104740 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.104901 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.105648 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.105870 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 00:59:54.106858 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.087798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34113 (local address 127.0.0.1:45682)
0929 00:59:54.088590 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.088634 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:54.090479 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:54.091198 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:54.091205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:54.091722 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:54.092211 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.092226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.093688 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.093695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.095589 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.095601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.096607 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.096612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.096769 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.097418 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:54.097452 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:54.099831 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:54.103057 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.103064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.103085 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.103531 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.103961 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.103965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.103967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.104220 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.104884 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:54.104890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:54.105280 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 00:59:54.105505 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 00:59:54.105834 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":386,"threads_started":1}
I20260929 00:59:54.108628 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:54.109608 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.078430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.97.60:33635)
0929 00:59:54.089475 (+ 11045us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.089480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.093040 (+ 3560us) server_negotiation.cc:407] Connection header received
0929 00:59:54.093245 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.093254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.093465 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.093640 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.095321 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.096778 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.098461 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.099016 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.099152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.103262 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.103272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.103282 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.103321 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.106724 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.107552 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.107559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.107562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.107705 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.108144 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.108147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.108149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.108466 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.108776 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.109041 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.109216 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10818,"thread_start_us":10565,"threads_started":1}
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 00:59:54.122859 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.109441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33857 (local address 127.0.0.1:57054)
0929 00:59:54.110670 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.110694 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:54.110815 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:54.111120 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:54.111124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:54.111564 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:54.111770 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.111782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.112978 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.112983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.113885 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.113899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.114588 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.114594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.114755 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.115332 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:54.115348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:54.117533 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:54.120678 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.120682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.120684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.121004 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.121407 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.121410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.121412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.121545 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.122152 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:54.122158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:54.122319 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 00:59:54.122430 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 00:59:54.122608 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112}
I20260929 00:59:54.123188 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.109852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.0.97.61:33857)
0929 00:59:54.110019 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.110023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.110766 (+ 743us) server_negotiation.cc:407] Connection header received
0929 00:59:54.110923 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.110927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.110986 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.111062 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.111935 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.112802 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.114026 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.114409 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.114511 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.117730 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.117734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.117737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.117770 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.120560 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.121121 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.121131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.121134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.121285 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.121659 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.121664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.121675 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.122028 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.122174 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.122574 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.122807 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:59:54.124154 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
I20260929 00:59:54.126331 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.112361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.0.97.62:34113)
0929 00:59:54.112559 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.112564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.112606 (+ 42us) server_negotiation.cc:407] Connection header received
0929 00:59:54.112720 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.112723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.112775 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.112846 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.113938 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.114839 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.116201 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.116728 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.116826 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.120558 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.120564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.120568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.120607 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.123734 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.124434 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.124438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.124440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.124614 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.124991 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.124996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.124999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.125330 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.125614 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.125943 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.126090 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 00:59:54.137465 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.124896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60828)
0929 00:59:54.125140 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.125154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:54.125230 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:54.125570 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:54.125576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:54.126045 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:54.126213 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.126223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.127533 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.127556 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.128698 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.128713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.129717 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.129720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.129860 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.130369 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:54.130385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:54.132384 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:54.135209 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.135215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.135220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.135596 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.135958 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.135964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.135968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.136133 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.136663 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:54.136670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:54.136842 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 00:59:54.136961 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 00:59:54.137202 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260929 00:59:54.137751 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.124987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.97.60:33635)
0929 00:59:54.125137 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.125144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.125195 (+ 51us) server_negotiation.cc:407] Connection header received
0929 00:59:54.125301 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.125307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.125391 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.125542 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.126534 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.127408 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.128830 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.129579 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.129722 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.132599 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.132605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.132609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.132645 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.135075 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.135713 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.135721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.135727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.135850 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.136245 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.136248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.136250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.136477 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.136656 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.137082 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.137355 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":34}
I20260929 00:59:54.142920 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.127857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.0.97.61:33857)
0929 00:59:54.128006 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.128011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.128027 (+ 16us) server_negotiation.cc:407] Connection header received
0929 00:59:54.128217 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.128222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.128287 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.128384 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.129906 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.130583 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.132012 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.132418 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.132567 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.136903 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.136908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.136911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.136957 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.140289 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.140925 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.140931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.140935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.141081 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.141435 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.141441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.141445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.141801 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.142058 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.142320 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.142596 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 00:59:54.143921 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: Bootstrap starting.
I20260929 00:59:54.147516 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 00:59:54.147851 23360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:54.149120 23360 log.cc:824] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:54.151822 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: No bootstrap required, opened a new log
I20260929 00:59:54.150570 23187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "355a2ac63aab484c8000e4ac4a8cc288"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45688
I20260929 00:59:54.158838 23360 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.159157 23360 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd02c7fcc6774b98b6e097f95cb85c6f, State: Initialized, Role: FOLLOWER
I20260929 00:59:54.159623 23360 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.160615 23366 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.160981 23366 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.161686 23360 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 00:59:54.170908 23377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 00:59:54.171023 23377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 00:59:54.168926 23293 master_runner.cc:186] Error getting master registration for 127.0.97.62:34113: OK, instance_id { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" instance_seqno: 1790643593070783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:59:54.180006 23293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:54.205762 23212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 4
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:59:54.221778 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.206566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60830)
0929 00:59:54.206772 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.206793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:54.206940 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:54.207344 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:54.207348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:54.207753 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:54.207913 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.207922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.209185 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.209194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.210572 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.210584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.211363 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.211368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.211581 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.212231 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:54.212249 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:54.214788 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:54.218849 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.218856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.218861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.219398 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.219795 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.219800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.219804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.219987 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.220706 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:54.220712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:54.220882 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 00:59:54.221046 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 00:59:54.221364 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 00:59:54.221778 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.206658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.97.60:33635)
0929 00:59:54.206839 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.206846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.206868 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:59:54.207115 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.207121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.207196 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.207303 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.208069 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.208984 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.210711 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.211200 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.211351 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.215011 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.215018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.215022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.215068 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.218704 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.219522 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.219527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.219531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.219676 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.220097 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.220102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.220105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.220573 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.220761 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.221147 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.221382 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:59:54.227774 23212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: Bootstrap starting.
I20260929 00:59:54.231877 23212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:54.233039 23212 log.cc:824] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:54.235785 23212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: No bootstrap required, opened a new log
I20260929 00:59:54.241482 23212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.241889 23212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355a2ac63aab484c8000e4ac4a8cc288, State: Initialized, Role: FOLLOWER
I20260929 00:59:54.242434 23212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.243460 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.243835 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.244705 23212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:54.244804 23187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 00:59:54.245012 23187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:54.248534 23187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.251044 23335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355a2ac63aab484c8000e4ac4a8cc288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f"
I20260929 00:59:54.251541 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:54.252463 23187 leader_election.cc:288] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [CANDIDATE]: Term 1 election: Requested vote from peers 6755fdc66c314a8798205e70f3239bbc (127.0.97.61:33857), bd02c7fcc6774b98b6e097f95cb85c6f (127.0.97.60:33635)
I20260929 00:59:54.255257 23335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355a2ac63aab484c8000e4ac4a8cc288 in term 1.
I20260929 00:59:54.255873 23161 leader_election.cc:302] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355a2ac63aab484c8000e4ac4a8cc288, bd02c7fcc6774b98b6e097f95cb85c6f; no voters:
I20260929 00:59:54.256307 23380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 00:59:54.260766 23380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [term 1 LEADER]: Becoming Leader. State: Replica: 355a2ac63aab484c8000e4ac4a8cc288, State: Running, Role: LEADER
I20260929 00:59:54.261355 23380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.269522 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355a2ac63aab484c8000e4ac4a8cc288. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.270036 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:54.285493 23393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 00:59:54.290889 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } attempt: 5
W20260929 00:59:54.302631 23314 tablet.cc:2369] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 00:59:54.312636 23166 tablet.cc:2369] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 00:59:54.316293 23393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 00:59:54.319608 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.287493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.0.97.60:33635)
0929 00:59:54.287805 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.287811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.287979 (+ 168us) server_negotiation.cc:407] Connection header received
0929 00:59:54.288534 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.288547 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.288651 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.288740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.293014 (+ 4274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.294463 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.298267 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.299180 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.299307 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.306717 (+ 7410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.306727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.306735 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.306788 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.311698 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.313679 (+ 1981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.313717 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.313729 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.314131 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.315104 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.315114 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.315120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.315980 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.316699 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.317892 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.318568 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 00:59:54.326120 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.286972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.0.97.61:33857)
0929 00:59:54.287158 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.287165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.301110 (+ 13945us) server_negotiation.cc:407] Connection header received
0929 00:59:54.301401 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.301410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.301506 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.301584 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.304906 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.306137 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.308324 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.309097 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.309214 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.319480 (+ 10266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.319486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.319490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.319535 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.323253 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.324049 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.324054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.324057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.324248 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.324686 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.324692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.324695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.324976 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.325252 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.325535 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.325763 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 00:59:54 dist-test-slave-pzsx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 00:59:54.327647 23145 master_runner.cc:186] Error getting master registration for 127.0.97.61:33857: OK, instance_id { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" instance_seqno: 1790643593568158 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 00:59:54.352105 23145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 00:59:54.355829 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.291403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33635 (local address 127.0.0.1:60862)
0929 00:59:54.291628 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 00:59:54.291646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 00:59:54.291761 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 00:59:54.316893 (+ 25132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 00:59:54.316899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 00:59:54.317412 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 00:59:54.317604 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.317616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.332786 (+ 15170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.332793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.334266 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 00:59:54.334281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.340665 (+ 6384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.340680 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 00:59:54.340905 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.341582 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 00:59:54.341605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 00:59:54.344559 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 00:59:54.352282 (+ 7723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.352290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.352294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.352876 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.353431 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 00:59:54.353436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 00:59:54.353440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 00:59:54.353647 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 00:59:54.354740 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 00:59:54.354745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 00:59:54.354928 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 00:59:54.355135 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 00:59:54.355427 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 00:59:54.357044 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.292017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.0.97.60:33635)
0929 00:59:54.312214 (+ 20197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.312221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.312246 (+ 25us) server_negotiation.cc:407] Connection header received
0929 00:59:54.312300 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.312305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.312376 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.317768 (+ 5392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.318064 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.319368 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.337782 (+ 18414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.338490 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.341270 (+ 2780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.344823 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.344830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.344835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.344871 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.351889 (+ 7018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.353078 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.353113 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.353116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.353298 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.353798 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.353803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.353806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.354341 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.356261 (+ 1920us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.356392 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.356635 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20045,"thread_start_us":20452,"threads_started":1}
I20260929 00:59:54.357360 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.287337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.0.97.62:34113)
0929 00:59:54.287663 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.287680 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.320739 (+ 33059us) server_negotiation.cc:407] Connection header received
0929 00:59:54.323127 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.323134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.323226 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.323343 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.335824 (+ 12481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.337109 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.340079 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.340846 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.340952 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.346885 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 00:59:54.346892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 00:59:54.346897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 00:59:54.346937 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 00:59:54.350869 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.352158 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.352163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.352167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.352477 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 00:59:54.354228 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 00:59:54.354234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 00:59:54.354237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 00:59:54.354871 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 00:59:54.355349 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.356310 (+ 961us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.356748 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260929 00:59:54.362926 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: Bootstrap starting.
I20260929 00:59:54.367924 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: Neither blocks nor log segments found. Creating new log.
I20260929 00:59:54.369205 23286 log.cc:824] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: Log is configured to *not* fsync() on all Append() calls
I20260929 00:59:54.378378 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: No bootstrap required, opened a new log
I20260929 00:59:54.383004 23335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [term 1 FOLLOWER]: Refusing update from remote peer 355a2ac63aab484c8000e4ac4a8cc288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:54.384011 23380 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:54.393067 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.393440 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6755fdc66c314a8798205e70f3239bbc, State: Initialized, Role: FOLLOWER
I20260929 00:59:54.393967 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } }
I20260929 00:59:54.401435 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.401814 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.403441 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 00:59:54.403519 23261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355a2ac63aab484c8000e4ac4a8cc288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6755fdc66c314a8798205e70f3239bbc"
I20260929 00:59:54.404027 23261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 00:59:54.407851 23261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355a2ac63aab484c8000e4ac4a8cc288 in term 1.
I20260929 00:59:54.408864 23260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [term 1 FOLLOWER]: Refusing update from remote peer 355a2ac63aab484c8000e4ac4a8cc288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 00:59:54.410809 23380 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 00:59:54.417498 23366 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355a2ac63aab484c8000e4ac4a8cc288. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.417821 23366 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.422701 23406 mvcc.cc:205] Tried to move back new op lower bound from 7334476162563624960 to 7334476162100412416. Current Snapshot: MvccSnapshot[applied={T|T < 7334476162563624960}]
I20260929 00:59:54.423601 23366 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.423821 23366 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.433458 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355a2ac63aab484c8000e4ac4a8cc288. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.433792 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.442577 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.443034 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:54.443377 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.443666 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288 [sys.catalog]: This master's current role is: LEADER
I20260929 00:59:54.446622 23393 catalog_manager.cc:1347] Generated new cluster ID: 61c5666f3e1f4d80a28617142c9d12dc
I20260929 00:59:54.446731 23393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 00:59:54.448856 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355a2ac63aab484c8000e4ac4a8cc288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a2ac63aab484c8000e4ac4a8cc288" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34113 } } peers { permanent_uuid: "6755fdc66c314a8798205e70f3239bbc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33857 } } peers { permanent_uuid: "bd02c7fcc6774b98b6e097f95cb85c6f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33635 } } }
I20260929 00:59:54.449092 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 00:59:54.468539 23393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 00:59:54.470526 23393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 00:59:54.494303 23393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355a2ac63aab484c8000e4ac4a8cc288: Generated new TSK 0
I20260929 00:59:54.495404 23393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 00:59:54.519855 23425 catalog_manager.cc:1259] Loaded cluster ID: 61c5666f3e1f4d80a28617142c9d12dc
I20260929 00:59:54.520071 23425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: loading cluster ID for follower catalog manager: success
I20260929 00:59:54.524019 23425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: acquiring CA information for follower catalog manager: success
I20260929 00:59:54.526800 23425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6755fdc66c314a8798205e70f3239bbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 00:59:54.634356 23214 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.614040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.97.62:34113)
0929 00:59:54.614261 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.614268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.631395 (+ 17127us) server_negotiation.cc:407] Connection header received
0929 00:59:54.632762 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.632768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.632857 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.632933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.634084 (+ 1151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39824: BlockingRecv error: recv got EOF from 127.0.0.1:39824 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:59:54.636592 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.615866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.0.97.60:33635)
0929 00:59:54.616055 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.616061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.632261 (+ 16200us) server_negotiation.cc:407] Connection header received
0929 00:59:54.635060 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.635066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.635175 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.635242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.636346 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39460: BlockingRecv error: recv got EOF from 127.0.0.1:39460 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 00:59:54.639611 23290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.629694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.97.61:33857)
0929 00:59:54.629907 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.629914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.637457 (+ 7543us) server_negotiation.cc:407] Connection header received
0929 00:59:54.638383 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.638389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.638472 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.638540 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.639387 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38580: BlockingRecv error: recv got EOF from 127.0.0.1:38580 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 00:59:54.685667 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.670563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.0.97.61:33857)
0929 00:59:54.670759 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.670766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.677552 (+ 6786us) server_negotiation.cc:407] Connection header received
0929 00:59:54.678113 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.678120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.678395 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.678544 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.679226 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.680404 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.682538 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.683198 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.683327 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.684078 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.684826 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.684944 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.685176 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.685270 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 00:59:54.732013 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.718852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.0.97.60:33635)
0929 00:59:54.719038 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.719044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.729825 (+ 10781us) server_negotiation.cc:407] Connection header received
0929 00:59:54.730386 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.730393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.730528 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.730618 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.731720 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39462: BlockingRecv error: recv got EOF from 127.0.0.1:39462 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:59:54.781931 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.767970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.0.97.61:33857)
0929 00:59:54.768192 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.768199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.775078 (+ 6879us) server_negotiation.cc:407] Connection header received
0929 00:59:54.775562 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.775570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.775662 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.775725 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.776439 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.777639 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.779732 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.780434 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.780621 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.781074 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.781243 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.781336 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.781516 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.781599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 00:59:54.792560 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.790450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.0.97.60:33635)
0929 00:59:54.790625 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.790631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.791108 (+ 477us) server_negotiation.cc:407] Connection header received
0929 00:59:54.791287 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.791293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.791390 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.791500 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.792304 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39464: BlockingRecv error: recv got EOF from 127.0.0.1:39464 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 00:59:54.804607 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.797661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.97.61:33857)
0929 00:59:54.797824 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.797830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.798279 (+ 449us) server_negotiation.cc:407] Connection header received
0929 00:59:54.798444 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.798450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.798541 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.798630 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.799076 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.800279 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.802205 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.802854 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.803001 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.803456 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.803591 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.803694 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.803850 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.803924 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 00:59:54.813692 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.811669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.0.97.60:33635)
0929 00:59:54.811843 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.811849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.811989 (+ 140us) server_negotiation.cc:407] Connection header received
0929 00:59:54.812255 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.812262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.812372 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.812480 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.813436 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39480: BlockingRecv error: recv got EOF from 127.0.0.1:39480 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 00:59:54.828040 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.821347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.0.97.61:33857)
0929 00:59:54.821577 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.821583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.822119 (+ 536us) server_negotiation.cc:407] Connection header received
0929 00:59:54.822287 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.822293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.822384 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.822484 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.822875 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.823992 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.825900 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.826620 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.826744 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.827197 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.827352 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.827449 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.827610 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.827698 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260929 00:59:54.838598 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.834541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.0.97.60:33635)
0929 00:59:54.834747 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.834755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.837062 (+ 2307us) server_negotiation.cc:407] Connection header received
0929 00:59:54.837227 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.837234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.837332 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.837450 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.838354 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39482: BlockingRecv error: recv got EOF from 127.0.0.1:39482 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 00:59:54.863588 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.854081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.97.61:33857)
0929 00:59:54.854278 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.854285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.857071 (+ 2786us) server_negotiation.cc:407] Connection header received
0929 00:59:54.857301 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.857307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.857410 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.857530 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.858023 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.859214 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.861232 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.861932 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.862081 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.862677 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.862829 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.862926 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.863096 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.863221 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 00:59:54.872970 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.871113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.0.97.60:33635)
0929 00:59:54.871302 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.871307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.871329 (+ 22us) server_negotiation.cc:407] Connection header received
0929 00:59:54.871631 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.871636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.871728 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.871829 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.872724 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39488: BlockingRecv error: recv got EOF from 127.0.0.1:39488 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 00:59:54.899652 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.892644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.97.61:33857)
0929 00:59:54.892841 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.892848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.893400 (+ 552us) server_negotiation.cc:407] Connection header received
0929 00:59:54.893596 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.893603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.893694 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.893810 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.894291 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.895538 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.897514 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.898190 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.898336 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.898825 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.898960 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.899069 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.899248 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.899370 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 00:59:54.917076 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.914661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.0.97.60:33635)
0929 00:59:54.914862 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.914869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.915429 (+ 560us) server_negotiation.cc:407] Connection header received
0929 00:59:54.915649 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.915656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.915771 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.915895 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.916821 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39500: BlockingRecv error: recv got EOF from 127.0.0.1:39500 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 00:59:54.959908 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:54.952757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.0.97.61:33857)
0929 00:59:54.952976 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.952983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.953539 (+ 556us) server_negotiation.cc:407] Connection header received
0929 00:59:54.953783 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.953790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.953896 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.954014 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:54.954468 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.955663 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.957721 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:54.958455 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:54.958590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:54.959079 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:54.959211 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:54.959299 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:54.959501 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 00:59:54.959588 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 00:59:54.968930 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:54.967080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.0.97.60:33635)
0929 00:59:54.967246 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:54.967251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:54.967434 (+ 183us) server_negotiation.cc:407] Connection header received
0929 00:59:54.967653 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:54.967659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:54.967748 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:54.967848 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:54.968712 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39504: BlockingRecv error: recv got EOF from 127.0.0.1:39504 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 00:59:55.041690 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.036727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.97.61:33857)
0929 00:59:55.036902 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.036906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.037282 (+ 376us) server_negotiation.cc:407] Connection header received
0929 00:59:55.037430 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.037434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.037506 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.037594 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.037993 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.038774 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.040108 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.040624 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.040731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.041046 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.041174 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.041263 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.041424 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.041483 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 00:59:55.049332 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 00:59:55.047441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.97.60:33635)
0929 00:59:55.047659 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.047666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.047934 (+ 268us) server_negotiation.cc:407] Connection header received
0929 00:59:55.048130 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.048136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.048214 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.048353 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 00:59:55.049120 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39520: BlockingRecv error: recv got EOF from 127.0.0.1:39520 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 00:59:55.173089 23377 catalog_manager.cc:1259] Loaded cluster ID: 61c5666f3e1f4d80a28617142c9d12dc
I20260929 00:59:55.173244 23377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: loading cluster ID for follower catalog manager: success
I20260929 00:59:55.175706 23377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: acquiring CA information for follower catalog manager: success
I20260929 00:59:55.177584 23377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd02c7fcc6774b98b6e097f95cb85c6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 00:59:55.184986 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.180446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.0.97.61:33857)
0929 00:59:55.180648 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.180652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.181016 (+ 364us) server_negotiation.cc:407] Connection header received
0929 00:59:55.181173 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.181177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.181243 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.181330 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.181670 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.182330 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.183596 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.183985 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.184073 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.184386 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.184480 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.184613 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.184717 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.184776 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 00:59:55.195565 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.190596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.0.97.60:33635)
0929 00:59:55.190828 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.190835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.190998 (+ 163us) server_negotiation.cc:407] Connection header received
0929 00:59:55.191113 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.191117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.191246 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.191339 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.191684 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.192365 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.193631 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.194028 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.194117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.194536 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.195020 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.195131 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.195195 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.195254 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 00:59:55.210079 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.201032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.97.62:34113)
0929 00:59:55.202470 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.202479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.203172 (+ 693us) server_negotiation.cc:407] Connection header received
0929 00:59:55.203444 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.203452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.203701 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.203895 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.204475 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.205482 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.207500 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.207935 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.208064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.208661 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.209088 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.209208 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.209660 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.209751 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":618,"threads_started":1}
I20260929 00:59:55.210523 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.201242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.97.61:33857)
0929 00:59:55.201484 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.201493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.203925 (+ 2432us) server_negotiation.cc:407] Connection header received
0929 00:59:55.204818 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.204824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.204929 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.205051 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.205486 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.206488 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.208426 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.209078 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.209197 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.209778 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.209896 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.209954 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.210186 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.210270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 00:59:55.211051 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 00:59:55.202126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.0.97.60:33635)
0929 00:59:55.202334 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 00:59:55.202342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 00:59:55.204568 (+ 2226us) server_negotiation.cc:407] Connection header received
0929 00:59:55.204947 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 00:59:55.204953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 00:59:55.205026 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 00:59:55.205131 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 00:59:55.206174 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.207223 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.209132 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 00:59:55.209725 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 00:59:55.209887 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 00:59:55.210290 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 00:59:55.210428 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 00:59:55.210489 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 00:59:55.210729 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 00:59:55.210813 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 00:59:55.215322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 23145
I20260929 00:59:55.216215 23207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:55.325712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 23145
I20260929 00:59:55.346333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 23219
I20260929 00:59:55.347506 23281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:55.455226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 23219
I20260929 00:59:55.475503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu pid 23293
I20260929 00:59:55.476809 23355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 00:59:55.583055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiPz2y/build/asan/bin/kudu with pid 23293
2026-09-29T00:59:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4296 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8833 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357696 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST